Now that the dust has settled on the 2019 Rugby World Cup, we can look back and analyze the accuracy of the World Rugby rating system. Before each World Cup, a great deal of attention is paid to the numerical ratings and the resulting ordinal ranking positions of the World Rugby system, and the ratings are assumed to provide guidance on what to expect in terms of match outcomes.
But just how accurate are those ratings? How often do higher-rated teams defeat lower-rated teams? Huge expectations are attached to the four top-rated teams prior to the World Cup, so how many of the top-rated teams actually found themselves among the final four during this competition and in competitions past?
Predictive accuracy of World Rugby ratings
Our analysis of predictive accuracy is based on the proportion of games won by the favourite team, with the favourite determined according to pre-competition ratings. For this analysis we have ignored canceled games. During World Cups, cancelled games are treated as draws, and to avoid having to use an arbitrary value for drawn games, accuracy is tabulated only for games with a winner and loser.
Table 1 summarizes the predictive accuracy of the World Rugby rating system for the five World Cups contested from 2011 to 2019, three for the men’s game and two for the women’s game. Overall, the higher-rated team won a remarkable 88% of the games, with predictive accuracy for the men’s game being slightly higher than for the women’s game. Overall accuracy was 91% for the group phase (during which each team plays every other team in their assigned group) and 79% for the knockout phase (played by the winners and runners-up of each group). In the group phase, accuracy for the women’s game was higher than for the men’s (an almost perfect 97% versus 90%), but lower for the knockout phase (75% compared to 83%). That data might suggest that the women’s game has less competitive balance in the group phase; however, after the weaker teams have been eliminated, the remaining teams appear to have better competitive balance.
TABLE 1 Predictive accuracy of the World Rugby rating system for the last five Rugby World Cups.
| Group matches | Knockout matches | All matches | |||||||
| Year | G | P(%) | G | P(%) | G | W | D | L | P(%) |
| Men | |||||||||
| 2011 | 40 | 90 | 8 | 100 | 48 | 43 | 1 | 4 | 92 |
| 2015 | 40 | 90 | 8 | 88 | 48 | 43 | 0 | 5 | 90 |
| 2019 | 40 | 89 | 8 | 62 | 48 | 38 | 3 | 7 | 84 |
| All | 120 | 90 | 24 | 85 | 144 | 124 | 4 | 16 | 89 |
| Women | |||||||||
| 2014 | 18 | 94 | 12 | 83 | 30 | 26 | 1 | 3 | 90 |
| 2017 | 18 | 100 | 12 | 67 | 30 | 26 | 0 | 4 | 87 |
| All | 36 | 97 | 24 | 75 | 60 | 52 | 1 | 7 | 88 |
| All | 156 | 91 | 48 | 79 | 204 | 176 | 5 | 23 | 88 |
Legend: G = games played, W = win by favorite, D = draw, L =loss by a favorite, P = prediction accuracy with draws not included.
Next, our analysis compares the four top-rated teams pre-competition to the four teams finishing top in each of the five World Cups. Table 2 shows that for each competition, three out of the four pre-competition favourites made it through to the semifinal stage: a success rate for the rating system of 15 out of 20, or 75%. However, the success rate is actually higher when you adjust for the luck (or perhaps we should say “the bad luck”) of the draw. In three of the five World Cups, in the last round before the semifinals, two of the four pre-competition favourites were paired up, meaning that only a maximum of three favourites could advance to the final stages of the competition. The adjusted success rate then becomes 15 out of 17, or 88% – coincidentally, the same as the game-by-game predictive accuracy.
TABLE 2 The top four teams in five recent World Cups, and each team’s pre-competition (pre-WC) ranked position.
| World Cup | Men 2011 |
Men 2015 |
Men 2019 |
Women 2014 |
Women 2017 |
|||||
| Team | Pre-WC | Team | Pre-WC | Team | Pre-WC | Team | Pre-WC | Team | Pre-WC | |
| 1 | New Zealand | 1 | New Zealand | 1 | S. Africa | 4 | England | 2 | New Zealand | 2 |
| 2 | France | 4 | Australia | 2 | England | 3 | Canada | 4 | England | 1 |
| 3 | Australia | 2 | S. Africa | 3 | New Zealand | 2 | France | 3 | France | 4 |
| 4 | Wales | 6 | Argentina | 8 | Wales | 5 | Ireland | 5 | USA | 7 |
A surprising criticism
The World Rugby rating system was purposely created by the former International Rugby Board with predictive accuracy in mind. Our analysis suggests that the system succeeds in that goal quite well.
So it was something of a surprise to read in August that World Rugby’s vice-president, Agustin Pichot, had dismissed the system as “ridiculous”. Pichot is reported to have said: “It is a ranking that is badly done and I said it the first day I arrived at World Rugby. It has no order, it is all mathematical and I would say that it is almost a matter of marketing.”
It is hardy “a matter of marketing” to have a rating system that predicts game winners 88% of the time. And while mathematics is involved, the system is hardly mathematically complex. In fact, the system can be described by one simple equation:
New rating = Old rating + Multiplier x (Actual – Predicted)
The multiplier recognizes only two score differences: up to 15 points and more than 15 points. (Exact score difference could be included to enhance accuracy, but an element of sportsmanship is involved here). The multiplier is also adjusted based on whether a game is a World Cup match or any other full international. In total, then, there are four values for the multiplier: 2 for an international decided by up to 15 points; 3 for an international decided by more than 15 points; 4 for a World Cup match decided by up to 15 points; and 6 for a World Cup match decided by more than 15 points.
The actual result is scaled on a 0-1 basis, with 0 for a loss, 0.5 for a draw and 1 for a win. The predicted outcome, meanwhile, (which must be limited to the range 0-1) is 0.5 + (rating difference) /20, and a home team gains three rating points before calculating the rating difference.
World Rugby’s explanation of the system is complicated, resorting to a number of plots. But we can use just one example from World Rugby’s web page to explain how the system works. Wales, with a pre-match rating of 76.92, is to play Scotland, with a pre-match rating of 76.36, at Cardiff, in a Six Nations international. The home team, Wales, gains three rating points, making the rating difference, from Wales’ perspective, 3.56. Now, suppose Wales plays a draw against Scotland. The actual outcome is 0.5 and the multiplier is 2. The predicted outcome is 0.5 + 3.56/20 = 0.68 (meaning Wales was 68% likely to win). Wales’ new rating, after the draw, is:
New Wales rating = 76.92 + 2 x (.5 – 0.68) = 76.92 – 0.36 = 76.56.
In dismissing the World Rugby ratings as “ridiculous”, Pichot said: “I’m going to change it, I assure you.” My advice, based on this analysis, is: “Don’t fix what isn’t broken.”
About the author
Ray Stefani is an emeritus professor of engineering at the California State University, Long Beach.
Editor’s note
For another Rugby World Cup post-mortem, check back in a few days to read Niven Winchester’s assessment of the Rugby Vision algorithm’s predictive performance.
Further reading
- Women’s Rugby World Cup results
- Men’s Rugby World Cup results
- Stefani, R.T. (2017) Six Nations Rugby – who’s the biggest overachiever?, significancemagazine.com
- Stefani, R.T. (2019) Six Nations rugby: Increased predictability (despite England upsets), significancemagazine.com
