In an earlier article, we analysed Rugby World Cup performance to find the overachievers (with Argentina leading the way) and the underachievers (with France being the major disappointment).
We defined overachievement to mean that a team produced more upset wins against higher-rated teams than they did upset losses against lower-rated teams. But before we begin analysing 6 Nations performance, we will first explain the World Rugby (formerly IRB) rating system.
The World Rugby rating system
World Rugby has done its homework in creating a well-designed rating system. There are three basic categories of rating systems: those that are subjective (as in boxing and the martial arts, where experts classify the athletes); those where points are accumulative (based on success but not on the opponent, as in WTA and ATP tennis); and those that are adjustive (in that the ratings self-adjust, based on the opponent and on the result of each competition, as in netball, cricket, women’s football and rugby). See Stefani (2010, 2014, 2015) for more details.
The adjustive category includes the most information and, as we would expect, has proven to be most accurate in practice. In Stefani (2015), we tabulated the results of 39 world championships involving 2,083 matches played in 13 sports, for both men and women. The higher rated team using adjustive systems won 84% of the non-tied matches, compared to only 73% for the accumulative systems. The adjustive World Rugby system was correct in 85% of world championship matches as we discussed in the article mentioned above.
The World Rugby system adjusts the ratings of match opponents by comparing the actual match result, A, calculated on a 0-1 scale (0 for a loss, 0.5 for a tie and 1 for a win) to a pre-match probability of winning, P, limited to a 0-1 scale, based on rating difference, d, adjusted for home advantage, where P is 0.5 (1 + d /10). The idea is to create neutral-field ratings. For a given match, three rating points are added to the home team rating so, for the home team, d is home team rating – away team rating + 3. The home team must then win a bit more often than if it had been playing away, to defend the home advantage. For the away team, d is of opposite sign. World Rugby calculates each new rating using a multiplier K, which affects adjustment size. The ratings adjust using the equation:
New Rating = Old Rating + K (A-P)
World Rugby selected K for predictive success. With too small a value for K, the system becomes insensitive to match performance, while too large a value would cause over-reaction followed by weeks of up-and-down adjustment. World Rugby chose K equal to 2 for most matches, 3 for a match decided by more than 15 points and twice those values for World Cup matches. The magnitude of the adjustment K (A-P) is the same for both teams in each match; but is always opposite in sign, causing the rating average to remain constant.
Typically, the best of all teams is about 70 rating points higher than the worst of all teams. That gap seems surprisingly large, given that a team only needs to be 10 rating points higher than an opponent to be 100% likely to win, according to the equation for P. In this study, the ratings of the 6 Nations before the first kickoff are used for each of the last six competitions, 2011-2016.
What benchmarks should be used to evaluate the success of the World Rugby rating system for the 6 Nations competition? The 85% predictive accuracy exhibited by the World Rugby system in world championship competitions would be an unreasonable expectation: the teams in the group phase of world championship matches are of widely varying ability, compared to the more challenging 6 Nations Rugby competition, which is played among the same six nations, who have become quite familiar with each adversary, much as would be true of any professional league competition. In Stefani (2014) the result of making over 20,000 match predictions in four sports, was that an adjustive rating system was 67% accurate in professional league competitions, which seems a reasonable benchmark for the 6 Nations competition.
Prediction accuracy and home advantage
In any given year of 6 Nations Rugby, each team plays each of the other five teams once, resulting in an imbalance for the number of home and away matches. However, in the following year, the home-away pattern is flipped. The result is that each pair of years forms a double round-robin, where each team plays each opponent twice, once at home and once away. Over a large number of such double round-robins, we would expect the average rating difference, home team rating – away team rating, to be close to zero, making d=3 for the proto-typical home team. That is, using that three-point home advantage, we should expect the typical home team to win P equal to 0.5 (1 + 3/10), or 65% of matches. Table 1 summarizes the success of the favourite (the higher rated team) and of the home nation for each year of the six years from 2011 to 2016, and then for three pairs of years. Ties are ignored in calculating success.
Table 1. Success of the Favourite and of the Home Team for 6 Nations Rugby, 2011-2016
| Favourite | Home | |||||||||
| Year | G | W | L | T | Correct (No Ties) | G | W | L | T | Home Win (No Ties) |
| 2011 | 15 | 10 | 5 | 0 | 0.667 | 15 | 9 | 6 | 0 | 0.600 |
| 2012 | 15 | 10 | 4 | 1 | 0.714 | 15 | 8 | 6 | 1 | 0.571 |
| 2013 | 15 | 6 | 8 | 1 | 0.429 | 15 | 9 | 5 | 1 | 0.643 |
| 2014 | 15 | 13 | 2 | 0 | 0.867 | 15 | 10 | 5 | 0 | 0.667 |
| 2015 | 15 | 12 | 3 | 0 | 0.800 | 15 | 7 | 8 | 0 | 0.467 |
| 2016 | 15 | 11 | 3 | 1 | 0.786 | 15 | 10 | 4 | 1 | 0.714 |
| 2011-12 | 30 | 20 | 9 | 1 | 0.690 | 30 | 17 | 12 | 1 | 0.586 |
| 2013-14 | 30 | 19 | 10 | 1 | 0.655 | 30 | 19 | 10 | 1 | 0.655 |
| 2015-16 | 30 | 23 | 6 | 1 | 0.793 | 30 | 17 | 12 | 1 | 0.586 |
| Total | 90 | 62 | 25 | 3 | 0.713 | 90 | 53 | 34 | 3 | 0.609 |
The favourite has been much more successful for the most recent three years compared to the earlier three. For example, there were 17 upsets during the first three years; but, there were less than half as many, eight upsets, for the more recent period. The emergence of consistently strong and weak teams had an interesting effect on the home-team success in 2015 and 2016. It appears that many weaker teams were playing at home in 2015, having a losing record – the worst of the six years. Meanwhile, the flipped schedule of 2016 resulted in stronger teams playing at home, creating the highest home-team success. However, when the years are paired, 2015/16 had the same average home-nation success as for 2011/12 when the two yearly averages were close together. For 2011/12 and 2015/16, the home success was below the expected value of 65%, based on World Rugby’s use of a three-point advantage, while for 2013/14, the home nation was slightly above the 65% norm. For all six years, the home nation won 61% of the non-tied games, close to the 65% assumed advantage. For statistical significance based on 90 matches, there would have to be about a 10% deviation to reject the null hypothesis that three points (65%) is the correct home advantage. The favourite won a respectable 71% of the non-tied matches.
We can now examine the 25 upset wins and 25 upset losses over the six years of matches to identify the overachievers and underachievers.
Overachievement and underachievement by nation
Table 2 shows the distribution of upset wins and loses by nation, as well as the overall record of each nation. Ties are not used in the calculations. Wales was the biggest overachiever, scoring 11 upset wins to only two upset loses (a +9 measure of overachievement) for their 29 non-tied games. Thus, they secured an upset in 38% of those 29 games. At the opposite side of our spectrum, France exhibited an unenviable talent for snatching defeat from the jaws of victory with eight upset losses to only two upset wins (a -6 balance). France was the biggest underachiever in recent Rugby World Cups also. Had both France and Wales performed in non-tied games exactly as the ratings had suggested, France would have had a 19-9 win-loss record (with two ties) to Wales’ 13-16 record (with one tie).
Table 2. Upset Wins, Upset Loses and Team Records for 6 Nations Rugby 2011-2016
| Country | UW -UL |
Upset Wins (UW) |
Upset Losses (UL) |
Matches As Expected |
Non-Tied Matches |
Fraction As Expected |
Wins | Losses |
| Wales | 9 | 11 | 2 | 16 | 29 | .552 | 22 | 7 |
| Italy | 3 | 4 | 1 | 25 | 30 | .833 | 5 | 25 |
| England | 1 | 4 | 3 | 23 | 30 | .767 | 25 | 5 |
| Ireland | -3 | 3 | 6 | 18 | 27 | .667 | 16 | 11 |
| Scotland | -4 | 1 | 5 | 24 | 30 | .800 | 6 | 24 |
| France | -6 | 2 | 8 | 18 | 28 | .643 | 13 | 15 |
| Total | 0 | 25 | 25 | 124 | 174 | .713 | 87 | 87 |
The other four teams had upset wins minus upset loss balances ranging +3 to -4. Italy and Scotland were the most predicable (80% or more) as they dutifully lost 25 and 24 times, respectively. England (77%) was both predictable and good. England had the best win-loss record at 25-5, with an upset win-loss balance of +1. Had they performed exactly as expected, their record would have had been almost the same at 24-6.
Summing up
The World Rugby rating system passed muster, in that higher rated pre-competition teams won 71% of the non-tied 6 Nations matches. Further, the home team won 61% of the non-tied matches; which is close to the expected success of 65%, given the three rating points that are added to the home-team rating. Wales was the biggest overachiever, pulling off an upset win 11 times (in 38% of non-tied games) with only two upset losses. Conversely, France disappointed as they did at the World Cup, gaining only two upset wins compared to eight upset losses. Two nations were predictably (80%) unsuccessful: Italy with 25 losses and Scotland with 24. Only England were predictably (77%) successful with 25 wins.
- Ray Stefani is an emeritus professor of engineering at California State University, Long Beach
References
- Stefani, R.T. (2010) A World of Sports and Rating Systems”, Proceedings of The Tenth Australasian Conference on Mathematics and Computers in Sport, July 5-7, 2010, Darwin, Australia. (http://www.anziam.org.au/Tenth+MCS).
- Stefani, R.T (2014) Results from 40 Years of Sports Analysis, Proceedings of the IACSS 2014 Conference, Darwin, Australia, 22-24 June, 2014.
- Stefani, R.T (2015) The Relative Competitive Balance of Male and Female Teams in World Championship Competition and the Relative Predictability of Official International Sports Rating Systems, Proceedings, Mathsport International, Mathsport 2015, Loughborough England, 29 June-1 July, 2015. (http://www.mathsportinternational.com/MathSport2015Proceedings.pdf)
