The Reliability and Accuracy of USGA Handicap Research II

(Note: This post revises a previous post based on comments by the USGA and a paper written by the USGA's Handicap Research Team.)

The USGA Handicap Section is responsible for how a golfer’s performance is measured and recommends how handicaps should be adjusted under different tournament formats.   Golfers for the most part have put blind faith in the USGA and the legitimacy of its recommendations.   It appears that such faith is misplaced.  The research behind the recommendations has never been vetted by outside academics,[1] is dated,[2]and in some cases just plain peculiar.[3]

The problems in USGA handicap research are illustrated by two tables prepared by the USGA on the probability of a player making an exceptional score.  The table presented in Appendix E of the USGA Handicap System is entitled “Odds of shooting an Exceptional Tournament Score.”  Part of that table is shown as Table 1 below.  

Table 1

Appendix E - Exceptional Tournament Score Probability Table[4]

Net Differential	0-5	6-12	13-21	22-30	>30
0	5	5	6	5	5
-1	10	10	10	8	7
-2	23	22	21	13	10
-3	57	51	43	23	15

 The values in the table are the odds of shooting a net differential EQUAL OR BETTER than the number in the left  column.

The definition of net differential appears to have changed over time leading to significant errors. The original research defines net differential as the difference between a player’s net score and the course rating.[5] The current definition of net differential in Appendix E is the handicap differential for a score minus the player’s Handicap Index.  The two definitions are different as can be seen by an example.   Assume a 17.0 index player scores a net 64 on a course with a slope rating of 150 and a course rating of 72.0.  The player is a 23 handicap and records a gross score of 87.  His net differential under the old system is -8.0.     His net differential under the current definition is -5.7 -- (87-72) 113/150 – 17.0.   Using the original definition of net differential, a player would have a 1 in 1138 chance of such a score or better.  Under the current definition of net differential, a player would have about a 1 in 300 chance of such a score or better. I suspect the old definition is the correct one, and the current Appendix E underestimates the rarity of low scores for courses with a Slope Rating greater than 113.  

The USGA shows some confusion about the distinction between odds and probabilities.[6]  The title of appendix references “probabilities,” but the body of the appendix is supposed to represent “odds.”   The USGA apparently does not know there is a difference between these two concepts.  “Odds against” is the ratio of the number of times an event does not occur to the number of times it does occur.  Probability is the ratio of the number of times an event occurs to the total number of trials.[7]  If the odds are 6:1, a golfer will have a net differential of zero or better 1 in 7 (not 6 as shown in Appendix E) rounds.  I assume, however, that the USGA meant probability and will accept the numbers in the table as that. 

Looking at Table 1, an inconsistency should be obvious.   If a player has an index between 13 and 21, he is 25 percent less likely to score a net differential of zero or better than a player with either a lower or higher index.  This is the only case where the probability of any level of performance (e.g., -2,-3…-10 net differential) does not increase or stay the same as a player’s index is increased.  Why is this index range cursed with such poor performance? I suspect any difference in probability among groups for scoring a net differential of zero or better may not be statistically significant.[8]  

Exactly how these probabilities were derived is not known.  The USGA does not have access to raw scores, only adjusted scores through its GHIN system.  Since adjusted scores are lower or equal to actual score, their use would overestimate the likelihood of an exceptional score.  If the USGA used actual scores, where did it get them, and how large was the sample?  Without a description of the research methodology, little credence should be given to the table.  

The table “Probability of Two Best Scores Beating Handicap” which appears on the USGA’s website presents more problems.  The table is shown as Table 2 below: 

Is the Table Accurate? – Table 2 represents the probability of scoring exactly 2 net differentials of various values.  This methodology underestimates the probability of exceptional scores.  The errors, however, are small in the range of interest of the USGA.    It also appears that Table 2 is based on Table 1.  Errors in Table 1 translate into errors in Table 2.  While Table 2 may be accurate, it is misleading.  For example, the table shows the probability of two net differentials of -6 is 1 in 3385.  The probability of two net differentials of -6 or better is 1 in 573—i.e., Table 2 decreases the probability by restricting the scores to a narrow range.  A more complete discussion of this issue is contained in the appendix.

Probability of Equivalent Pairs Are Not Equal – The probability of two best scores being (a,b) should be the same as the probability of (b,a).  This is the case for most of the pairs in Table 2.  There is an exception, however.  The probability of (6,9) is 17109.  The probability of (9,6) is 17189.  This is clearly just a typographical error made by the USGA, but such errors seem to plague the USGA handicap publications.[9] 

Table 2

Probability of Two Best Scores Beating Handicap 

	0	-1	-2	-3	-4	-5	-6	-7	-8	-9	-10
0	27	46	92	199	408	869	1808	2480	3871	9180	85779
-1	46	13	26	58	118	253	526	722	1126	2672	24987
-2	92	26	20	43	89	191	398	546	853	2023	18907
-3	199	58	43	59	121	258	537	737	1150	2728	25492
-4	408	118	89	121	200	427	888	1219	1903	4512	42163
-5	869	253	191	258	427	821	1708	2343	3667	8872	81030
-6	1808	526	398	537	888	1708	3385	4644	7249	17189	****
-7	2480	722	546	737	1219	2343	4644	6225	9716	23041	****
-8	3871	1126	853	1150	1903	3667	7249	9716	1492	35361	****
-9	9180	2672	2023	2728	4512	8672	17109	23041	35361	82951	****
-10	85779	24967	18907	25492	42163	81030	****	****	****	****	****

 Many of the questions surrounding USGA handicap research could be resolved if it were peer reviewed outside of the USGA.  The only plausible reason for the USGA’s reluctance to release research, however, is it knows it cannot withstand scrutiny. Once the curtain is pulled back, the USGA’s perceived omnipotence is lost.  This is would not be a bad thing for the game.  

Appendix: Probability of Having Two Best Net Differentials

The probability of a player having his two best two differentials equal –n in 20 rounds is found by the following equation:  
                                                          P(-n) =  (sum from k=2 to 20)(20!/((20-k)!k!)) x^k a^20-k

Where,

P(-n) = The probability the best two differentials out of 20 equal -n
x = Probability of a net differential of -n or higher (i.e., a worse score)
a = Probability to a net differential of –n
k = Number of –n differentials

The probability P(-n) covers the occurrences of 2,3,….20 differentials of –n in 20 rounds –i.e., even if a player has four rounds of with a net differential of –n, his two best differentials are still –n.

To find the probability of any particular net differential, we use the probabilities of scoring a net differential of various values or better.  These are taken from Appendix E of the USGA Handicap System and are shown in Table 1A below.[1] 

Table 1A

Probability of Net Differentials or Better (13- 21.9 Index Player) 

Net Differential	0	-1	-2	-3	-4	-5	-6	-7	-8	-9	-10
Probability	1/5	1/10	1/21	1/43	1/87	1/174	1/323	1/552	1/1138	1/3577	1/37000

 The probability of a net differential between 0 and -1 is the difference between the probability of 0 or better and 1 or better (i.e., 1/5 – 1/10 = 1/10).  We call this P(0).  The probabilities of other net differentials are determined in the same way and shown in Table 2A.

Table 2A

Probability of Net Differentials

Net Differential	0	-1	-2	-3	-4	-5	-6	-7	-8	-9	-10
Probability	.1000	.05238	.02436	.01176	.00575	.00265	.00128	.00093	.00060	.00025	na

 As an example, we estimate the probability of the two best differentials being equal to 0.  As shown in Table 1, the probability of a net differential higher than 0 is .8.  The probability of a differential of 0 is .1 (see Table 2A).  We can now use the equation for P(-n) to calculate the probability of two best scores being equal to 0 as shown in Table 3A.

Table 3A

Probability of k Scores with a Differential of 0 

k	Combinations	P(0)n·P(>0)20-n	Probability
2	190	.00018	.03427
3	1140	2.2518E-5	.02567
4	4845	2.81475E-6	.01363
5	15504	3.51844E-7	.00545
6	38760	1.15292E-8	.00044
7	77520	5.49756E-9	.00042
		Total	.07997

The probability of having two best net differentials of 0 is 1 in 12. This differs from the USGA‘s estimate (1 in 27) since that estimate is based on only the occurrence of exactly 2 net differentials of 0. [2] 

The size of any error in the USGA calculations can be seen in Table 4A which presents the Equation estimate, the Exactly 2 estimate, and the USGA estimates. 

Table 4A

Probability of Two Best Scores 

Two Best Scores	Equation	Exactly 2	USGA
0,0	12	29	27
-1,-1	9	13	13
-2,-2	18	21	20
-3,-3	54	58	59
-4,-4	189	186	200
-5,-5	818	831	821
-6.-6	3349	3375	3385
-7,-7	6210	6245	6225
-8,-8	14855	14908	14912
-9,-9	82671	82545	82951

 All of the estimates are virtually the same for pairs -2 and better. Since this is the region of most importance in spotting exceptional scores, the USGA formulation is adequate.  It may, however, overstate how rare exceptional scores are.

The USGA presents the probability of the two best scores out of 20 being equal to -6 is 1 in 3385. But, using the same methodology, the probability of a player having two best scores of -6 or better is 1 in 572 or about six times more likely than having scores exactly equal to -6.[3]  

The rarity the USGA finds is due in part to the interval it has chosen.  Assume for example, the USGA had decimal intervals.  The probability of a player having a net differential of -3 is the probability of a net differential between -3 and -3.1.  The probability of two best net differentials of -3 (making simplifying assumptions) with this shortened interval is about 1 in 6800.   Basing any modification of a player’s handicap on this probability would clearly not be fair. 

The better method is to find the probability of two scores of –n or better.  If that probability is found unreasonable, a player should have his index reduced.  In examining the Handicap Reduction Table,[4] it appears the probability of –n or better has been used.  For example, for a player with 20 tournaments and an average net differential of -6, there is no handicap reduction.  That is, the probability of scoring two net differentials of -6 or better is 1 in 570 and within the range of reasonability (where 1 in 3385 would not be). [5]

---

[1] The USGA Handicap System, 2012-2015, USGA, Far Hills, NJ, 2012, p.117.

[2] Knuth, loc. cit.

[3] The confusion about what these numbers mean is shown in a USGA Publication (How Well Should You Play?):

“For example, the odds of our example player with a course handicap of 14 beating it by eight strokes (-8 net) once is 1,138 to one.  Put another way, the average player posts 21 scores a year.  That means to score this well, assuming the Handicap Index is correct, would take 54 years of golf to do once.”

This is incorrect.  The player is expected to accomplish the feat once in 54 years, but that is different than saying the feat will be accomplished at the end of 54 years.  The player may well do it tomorrow.  There is a 17 percent chance he will accomplish this feat in the first ten years.  There is a 37 percent chance he will not accomplish the feat even in 54 years.

[4] USGA Handicap System, p. 81.

[5] Appendix E rates performance by negative net differentials (score differential - handicap index).  The Handicap Reduction Table uses positive net differentials (handicap – average of two differentials) in measuring performance. For consistency the USGA should use one type of differential.  It is also not clear why the Handicap Reduction Table goes up to an average net differential of -14 when such scores are highly unlikely according to Appendix E. 

---

[1] It is USGA policy not to release its research to any outside agency.

[2]  The USGA recommends a 90 percent allowance for four-ball stroke play. This allowance appears to come from a small study conducted by Francis Scheid, a one time member of the USGA Handicap Research Team, in 1971 (See F. Scheid, “You’re Not Getting Enough Strokes,” Golf Digest, Trumbull, Connecticut, June 1971, p. 52). In this study, Scheid took 50 scorecards from members of his home club and simulated matches among them. Using such a small sample drawn from non-tournament conditions makes any result suspect.  

[3] In four-ball stroke play, for example, men play to 90 percent of their handicap, and women play to 95 percent.  The USGA has never presented any theory on why the absence of a Y-Chromosome should alter a player’s handicap allowance.

[4] Note: Appendix E has been abbreviated here.  It actually shows probabilities up to a -10 net differential.

[5] Knuth, D. L., et. al., Outlier Identification Procedure for Reduction of Handicaps, Science and Golf II, E & F Spon, Chapman and Hall, London 1994, p.230.  This definition is affirmed in an article written by John Paul Newport in the Wall Street Journal (July 2, 2011).  Dean Knuth, former Senior Director of Handicapping at the USGA, is quoted saying the odds of a midrange player shooting eight strokes better than his course handicap are 1 in 1,138. 

[6] Since the first draft of this paper, the USGA has dropped “odds” and now uses “probabilities” in Appendix E.

[7] Goldberg, Samuel, Probability: An Introduction, Prentice Hall, Englewood Cliffs, N.J, 1960, p. 70.  Let E be any event and P(E) be the probability of that event.  We say that the odds for E are a to b  if and only if:

                P(E)=a/(a+b).  If odds for E are a to b, then the odds against E are b to a.

[8] Since the first draft of this paper, the probability for this 13 to 21 index range has been dropped from 1 in 6 to 1 in 5.  No explanation was given for the change. 

[9] This mistake is not an isolated example. For at least 5 years, Appendix E has shown the following incorrect equation:

                                                                    74 -71.2 = 2.8 x 113/126 = 2.5 Handicap Differential

The equation should be:

                                                    (74-71.2) X 113/Slope Rating = 2.8 x 113/126 = 2.5 Handicap Differential

 (Note: This error was corrected for the 2012-2015 edition of the Handicap System.  Unfortunately, current edition contains new errors).