Trading: All about Robustness Testing and reliability of an algorithm

Find out why and how robustness testing plays a crucial role in assessing the quality of an algorithmic trading strategy.

Robustness testing, or how to evaluate the reliability of an algorithmic trading strategy

A battery of statistical tests carried out in the most varied contexts possible, the robustness test allows the trader to ensure the efficiency and resilience of his algorithm, whatever the market conditions.

Although software such as ProRealTime makes the process of designing a trading algorithm much easier, it is still vital to test your trading programs methodically and accurately before entrusting them with the management of any of your financial assets.

The robustness test is an essential prerequisite for the successful launch of an automatic trading strategy and allows the trader to evaluate the performance of his algorithm and at the same time avoid the main pitfalls and statistical biases!

Robustness Test

What is robustness testing?

Robustness testing is a methodology that aims to determine the level of reliability of a system.

In the context of algorithmic trading, a robustness test is used to verify the performance, consistency and resilience of a trading algorithm. By doing so, the trader can then discard the most fragile strategies and favour the most robust ones.

Although a simple backtesting operation can allow the trader to simulate the performance of an algorithm over a given period and a given market, the robustness test goes further by evaluating the significance of the results obtained.

The objective is therefore a priori very simple since it is a question for the Trader of extracting the luck factor from the equation, in order to obtain the clearest possible vision of the value of his algorithm.

However, distinguishing a lucky guess from a real ability to beat the market can quickly become quite complex. This is where robustness testing comes into its own.

Although there are many robustness tests proposed by the scientific literature (White’s test, Hausman’s test, or Breusch-Pagan’s test), they are not always easy to apply in practice when evaluating an algorithmic trading strategy.

Let’s try to go through some examples of relatively simple robustness tests adapted to online algorithmic trading.

The importance of multiplying test scenarios

Whether they are carried out on historical prices (backtests) or under live conditions but on a simulation account, the tests of your trading strategies must be as numerous and varied as possible to counter the dangers of over-optimization.

Indeed, all things being equal, the more you increase the test period of a strategy, and the more you multiply the diversity of market scenarios encountered, the more significant the result of your test will be.

For example, rather than testing your strategy over a year, why not test it over a decade? Better still, if you have already tested your strategy over a long period, what if you now decide to test it only on even-numbered days? Or to run it only in the autumn and winter months?

As you can see, the idea here is to check that the results of your strategy remain relatively constant whatever the period tested. If the results are relatively stable, you can conclude that your strategy is relatively robust (based on this criterion), and conversely, if the performance of the strategy changes completely, you should be concerned about its reliability!

Finally, this principle of test diversification also applies to the assets on which you test your strategy. Rather than testing your algorithm only on the CAC 40, why not observe its behaviour on other world stock market indices?

NB: Also make sure that the market conditions encountered by your algorithm are varied. Think of including in your backtest periods bullish, bearish and “range” market phases, but also phases of low and high volatility.

Do not neglect sensitivity analysis

While varying the data samples is a particularly useful first test of the robustness of a trading strategy, it is strongly recommended to complement it with one or more sensitivity analyses.

Sensitivity analysis is, as the name suggests, a way of measuring the sensitivity of a trading strategy’s performance to changes in one or more of its variables.

For example, let’s say your strategy is programmed to automatically place a stop-loss order at a distance of 50 points and a take profit at a distance of 100 points.

What happens if you now test it with a stop-loss at 49 points and a take-profit at 101 points? Do the results remain the same or do they change dramatically?

The Monte Carlo method allows you to evaluate the sensitivity of your strategy accurately so that you can get a clear picture of how your algorithm will behave in a wide range of scenarios.

Monitor the right indicators

Finally, when carrying out your robustness tests, make sure you are not hypnotised by the performance of your algorithm alone. Indeed, in the financial markets, a performance only makes sense in the light of the risk taken to achieve it!

You will agree that a strategy with a performance of +50% one time out of ten is meaningless if it causes you to lose all your capital the rest of the time.

Consequently, it is important to follow other criteria such as the Profit Factor, the Maximum Drawdown, the number of consecutive losing trades, the ratio of winning trades to losing trades…

Above all, your robustness tests should allow you to reassure yourself on the fundamental points in your eyes, and allow you to select a strategy in accordance with your personal situation and your own objectives.

In conclusion, testing the robustness of a trading strategy is essential before entrusting it with any of your capital. Many strategies that look very attractive turn out to be particularly fragile when examined under the microscope.

Share this

  1. Kovit • 03/11/2021 #

    I haven’t looked in the blog section of the website for some time but certainly will make a point of checking it now, thank you for all your hard work Nicolas putting together and posting all this extra learning content.

  2. buffster76 • 03/11/2021 #

    Hi @Nicolas, I’ve been offline for a few months developing my own python scraper and backtester and now I’m back to see v11 is improved! I would love to see a different version of walkforward which would allow variables to be optimised depending on market conditions. For example, take a daily strat, optimised in 3month chunks over the last 10yrs, then for each chunk optimise and also characterise market trend, ATR, ADX etc and then look for correlations between strat variables and market conditions. I’ve done this semi-manually using python to characterise and it seems to improve the robustness of my code although I don’t have any proof yet to demonstrate this. Has this already been done in PRT? I think it would be quite simple to calculate market “characteristics” for each WF step?

    • Nicolas • 03/11/2021 #

      once the walk forward analysis is made, you know the different OOS+IS periods, therefore you are be able to analyze those periods with any calculation to characterize them. Now that we have variables arrays, it is easier than before!

  3. buffster76 • 03/11/2021 #

    variables arrays? I haven’t used these before, ok I will learn this next, many thanks

    • Nicolas • 03/11/2021 #

      Yes, easier but no simple 🙂 Good luck!

  4. David_NZ • 03/11/2021 #

    Hi, I am new here. I am also a newbie to forex and determined not to be in the 95% of newbies loosing their shirt. Hence this subject of great interest. My learning to date has been on MT5 and ESB. The ability to backtest on historical data on the terminal is excellent. The small print everywhere says high risk. The main message to come from that small print is tomorrows market may be different from ANY TIME BEFORE. Variety of backtest market is very important. 10yrs back what point is there in that. It still may not include tomorrows action. I am currently using Feb 2020 to May 2021 H4 [Tester]
    Expert=MarketEasy Strategy Builder 5.ex5 Symbol=GBPJPY Because this period has many market types, beginning of Covid-19 lock-down, UK split from EU and GBP JPY is noted for its volatility. Glad to find most of my thinking agrees with you Nicolas. Pitty I cannot copy code here into MQL5 editor and compile it.

  5. schneiderj • 148 days ago #

    Hello !

    Nicolas you said it would be good to use Monte Carlo analysis. But how we can do that with ProRealTime ? Is there an API to call it ?

    Thanks you

Register or