Strokes-Gained: Some Questions Asked and Answered

In a recent post,[1] it was argued that many of the findings in Mark Broadie’s book, Every Shot Counts,[2] were previously presented in a book written over 40 years ago by Alastair Cochran.[3]  Professor Broadie challenged that conclusion and made other criticisms of Cochran’s work.[4]  The purpose of this note is to examine three questions: 1) Are Broadie’s criticisms of Cochran’s work valid? 2) What are the true origins of the strokes-gained concept, and 3) Is the strokes-gained statistic either revolutionary or of value?  Each question is discussed in turn.

1. Are Broadie’s Criticisms of Cochran’s work valid? – Broadie’s criticisms are in italics followed by an analysis of their validity:

Even though Cochran and Stobbs title their chapter “Long Approach Shots-Where Tournaments are Won,” they actually present no evidence that this is the case.

It is true Cochran does not write that long iron play is the ultimate determinant of who wins.  That would be foolish, and Cochran is clearly not a fool.  Players win for a variety of reasons as Broadie’s own work shows.  Cochran merely argues  iron play is important and presents a table showing the difference in iron play between the top nine and bottom nine players (see Table 31:6 below).  Normalizing for the play of non-iron shots, Cochran concluded the top nine players had a five stroke advantage over the bottom nine players.  This is clearly evidence that iron play is important in winning.  To claim that  Cochran presents no evidence of the importance of iron play is clearly wrong.  Perhaps Cochran can be criticized for an overly definitive chapter title.  But you cannot argue Cochran did not identify the importance of long iron play long before the publication of Every shot Counts.

 

Table 31:6 Long Approaches at Birkdale: How the leaders compared with the tail-enders

		Distance from which shot is played (Yards)
		140-160	160-180	180-200	200-220
Median finishing distance from hole (yards)	Top nine players	9.8 (46)	9.0 (22)	13.6 (63)	14.5 (50)
	Bottom nine players	12.0 42	10.8 (17)	13.8 (64)	17.7 (54)

Broadie saves his most strident criticism for Table 31:8 that is reproduced below:

This is the main table in the chapter, on which the title is based.  It (Table 31:8) compares a 50% reduction in long approach shot errors (completely unrealistic given the 14% above) with “doubling the accuracy of putting” (also completely unrealistic) with hitting drives 20 yards further and having them all finish in the fairway (also completely unrealistic).  There are two problems with this.  First, the “strokes gained per round” from these assumptions is not explained and I’m pretty sure is not correct.  Second, even if we accept those numbers, it makes no sense to conclude anything from hypothetical, unrealistic assumptions that are not at all comparable.   It would be like saying the strokes gained per round from hitting drives 3 yards further is much less than one-putting twice as often, therefore putting is “most important.” 

Cochran and Stobbs (see below) write “we are not implying these improvements are equally easy to achieve” yet they still base their conclusions (“But it would not be too far to conclude”) solely on this one table with completely arbitrary assumptions.  It is crucial to base conclusions on comparable improvements; completely arbitrary assumptions lead to conclusions that are completely arbitrary.

Table 31:8 How much the pros at Birkdale would have gained by drastic improvement in their game

	Gain in Strokes per Round
By “doubling” accuracy of putting	4.2
By “doubling the accuracy of short approaches	1.7
By “doubling the accuracy of long approaches	5.5
By 100% accuracy and extra 20 yards on drives	2.2
Total	13.6

Broadie engages in sophistry in attempting to prove his points.[5]  Nowhere does Cochran maintain this table proves the primacy of iron play.  In fact, this table  is never cited in the text.  This would be strange if Cochran thought this to be his “main table.”  Cochran does conclude “it would not be going too far to conclude that it is in full iron play as well as putting that the main difference in caliber makes itself felt between different levels of top class professional golfers and the degree of success they have in tournaments.” [6]  His conclusion is based (I assume) on the study of difference in putting shown in Table 29:6 (Putting at Birkdale: How the leaders and the tail-enders compared with the field as a whole) and Table 31:6 shown above.   To argue that Cochran based his conclusion on Table 31:8 is a misstatement of the facts.

Cochran readily admits the improvements assumed in Table 31.8 are unrealistic.[7]   Broadie complains that not much can be drawn when unrealistic assumptions are made. Yet that does not stop Broadie from analyzing the impact of hitting the ball 20 yards further.[8]  He argues this assumption is unrealistic but it is important to understand “the trade-off between distance and accuracy for course strategy.”[9]  Cochran was merely using the same literary device as Broadie and is undeserving of criticism.

The gain in strokes per round shown Table 31:8 could represent estimates of the marginal value of improvement in each category.  The marginal value of improvement must be weighed against the marginal cost of that improvement to arrive at the optimum practice plan.  For the average player, Cochran concludes improving his iron play is probably the best use of his time.  This is not much different than Broadie’s finding “The biggest contributor to scoring?  Approach shots, which contribute 40% to the total strokes gained.”[10]

Broadie complains the methodology behind the “strokes-gained” calculation is not explained.    Cochran did show estimates for iron play in Table 31:7 shown below. 

   

Table 31:7 - The benefits of improved iron approaches: an estimate of strokes gained by halving their inaccuracy (Abridged)

Hole	Length of hole remaining after 250 yard drive	Shots to get down from this distance
		Normal Standard	Improved Standard
1	243	No long approach required
2	177	3.13	2.74
9	160	3.05	2.71
18	200	3.23	2.79
Total		43.84	38.39

To examine the benefits of improved iron approaches Cochran compares the hole scores for players with standard accuracy with a player who is 50 percent more accurate.  Broadie is correct that Cochran does not detail the method used to estimate the “shots to get down.” (I assume this was the book editor’s decision much as in Broadie’s book where any description of his simulation model is omitted.)  It is possible to speculate on possible methods and show that Cochran’s estimates are not too far off, if at all.  One possible method would be to first estimate the normal standard finishing distance from the hole.  Cochran has established that the medium finishing position of an iron shot is approximately 7.5% of the starting distance.  At hole 9, for example, the average finishing position would be 12 yards.  From the graph in Fig. 29.2, the average number of putts from 12 yards is 2.05.  Therefore the standard number of shots to get down from 160 yards is 3.05 (2.05 +1).

If a player improved his accuracy, his finishing distance would be 6 yards.  The number of putts to get down from this distance is approximately 1.82.   With improved iron play, the player would get down in 2.82 strokes.  Cochran, however, reports the player would get down in 2.71 strokes.  Cochran could have used a more sophisticated method where a player with an improved standard hit more greens and/or missed more bunkers.  In any case, Cochran’s estimates in Table 31:7 appear to be in the ballpark.  Contrary to Broadie’s assertion, Cochran does not base a conclusion on them. 

In summary, Broadie’s criticisms are not valid.  Cochran could have presented more detail about his work, but there does not appear to be any evidence of poor scholarship as Broadie asserts.

2. What are the True Origins of the Strokes-Gained statistic?” - In Every Stroke Counts, Professor Broadie takes credit for the “strokes-gained” concept of measuring player performance:

Strokes-gained is the name for this new way of measuring shot quality.  It uses the same unit-strokes – to calculate the skill of the many kinds of shots taken throughout each round of golf.  The origins of most new ideas can be traced to the earlier work of others, and strokes-gained is no exception.  The term owes its heritage to a brilliant applied mathematician of the mid 20^th century (Richard Bellman), and a grand theory he called “dynamic programming” developed at the dawn of the computer era….Using this technique, I (emphasis added) developed a way to compare golf shots and quantify a golfer’ skill.[11]  

I do not believe the lineage of “strokes-gained” goes back to the works of Richard Bellman.[12]  The reference to Bellman seems soley intended to give “strokes-gained” some mathematical credibility by association.[13] Stripped down to its bare essentials, Broadie had access to a big pile of data from which he calculated the average score to complete a hole as a function of distance.  You take the average score from Point A and subtract the average score from Point B.  The difference minus one is the strokes-gained for the shot.  This is not dynamic programming as Broadie implies, but rather an exercise in second grade arithmetic.  Broadie admits as much later in the book when he writes “Once you have access to the data base showing average strokes to hole out from any given distance, there’s no rocket science involved in calculating strokes-gained, just subtraction.”[14] 

I strongly believe Broadie owes an intellectual debt to Cochran.  Both Broadie and Cochran tried to identify the importance of each type of shot is isolation from the others.  Only Cochran did it first.  Broadie refined the methodology to make it player-specific while Cochran’s results were only tournament-specific.  A review of how each researcher found the strokes-gained will reveal the similarities and differences in their approaches.

Putting – Broadie has estimated the average number of putting strokes it takes to hole the ball for various distances from the hole.  The average number of strokes minus the number of actual strokes taken is the putting strokes gained or lost on that hole.  Cochran’s method is similar.  Due to data limitations he compares the top nine players with the bottom nine players in an attempt to explain the difference in performance.  Cochran makes some calculations about the distribution of the lengths of first putts “assuming that their play up to the green has been the same—which, of course, it wasn’t.”[15] Cochran then applies the probabilities of each group to make a putt of a certain length.  He found the top nine players gained about a tenth of a stroke per hole over the bottom nine players due to putting.  Cochran attempted to normalize the distance of the putt in order to isolate the value of putting.  This is the basic idea behind Broadie’s strokes-gained putting.

Long Approach Shots – Broadie has estimated the average number of shots it takes to hole a ball from various distances and positions.   The strokes gained from a long approach shot is calculated as follows:

  Strokes- Gained = Average Number of Strokes to Hole from Starting Position – (Average Number of                     Strokes to Hole from finishing position +1)

As discussed under the previous question, Cochran employs much the same method.  Cochran writes:

“Taking all eighteen of the players concerned to have hit the average drive of the whole field,  and to have played their short  approaches and putted to the average standard too, the team were able to show that the difference in standard of strokes from 140 to 220 yards gave the top nine an advantage of about one and a quarter strokes per round over the bottom.”[16]

ShotLink allows Broadie to be more precise.  He can take a player’s actual drive and long approach and estimate the strokes gained.   This allows Broadie to estimate the shots gained for each player.  Without the vast data of ShotLink, Cochran had to assume all players started from the average drive.  He then applied the long approach accuracy estimates of Table 31:6 for each group of players to determine where the long approach shot finished.  From the finishing positions the number of strokes needed to hole the ball was estimated.  The difference in the estimates of the top nine and bottom nine players was the estimate of the strokes gained by the better players.  Broadie’s and Cochran’s methods are remarkably similar in concept. 

Driving  - Broadie has used ShotLink data to estimate the shots needed to complete a hole from any distance and a variety of positions (fairway, rough, sand bunker).  He takes the estimate of the number of shots to complete the hole from the tee and subtracts the number of strokes to complete the hole from where the drive finishes plus one.   While Broadie never presents any evidence of the accuracy or inherent variability of these estimates, they do allow for a straightforward calculation of strokes-gained driving for each player.  Cochran was hampered by his small data set.  The difference in driving between the top nine and bottom nine players was only 7 yards.  Cochran, by taking into account calculations made for the other types of strokes, showed the top nine players gained a half a stroke per round by their driving over the bottom nine.  Cochran does not specify his calculations.  It can be assumed that he used the same method as for the other types of shot.  That is, the longer drives led to more accurate long approaches, which led to fewer putts and a lower average score.  Both researchers reached the same conclusion that when it comes to driving, i.e., length counted for more than accuracy.

Short Approaches – Both Broadie and Cochran have difficulty dealing with this shot.  Under Broadie’s method, the number of shots to finish a hole is a function of distance.  There may be so much variability in 20 foot chip shots, for example, that Broadie’s method needs more refinement before it can be implemented on the PGA Tour.[17]  Cochran found no significant difference in the short approaches of the top nine and bottom nine players.   Both groups had similar median finishing distances for their short approach shots so no strokes were gained. 

Broadie does not give Cochran credit for coining the term “strokes-gained.”  Instead that honor goes to unnamed colleagues at MIT.[18]  His only mention of Cochran comes in the acknowledgments where he writes:

When I started research into golf, I didn’t even remember that in high school I had picked up from the library the now classic book Search for the Perfect Swing.  The authors Cochran and Stobbs in the 1960s were the first ones to record and analyze individual shots.[19]

(That must have been one great high school library since it acquired the book some twelve years before it was published in the United States.)   Broadie does not give Cochran any credit for birthing the strokes-gained method.  Given the similarities in their research approach described above, Cochran deserves much more recognition than he is given.

3)  Is the strokes-gained statistic either revolutionary or have value?  - The subtitle of Broadie’s book is “Using the Revolutionary Strokes Gained Approach to Improve Your Golf Performance.”  A new statistic can be termed revolutionary if it changes the way the game is played like Sabermetrics did for baseball.[20]  Broadie presents no evidence that knowledge of this statistic has altered a player’s decision-making the way Sabermetrics changed managerial decisions in baseball.

The strokes-gained statistic could also be revolutionary if it produced information about player performance that was previously unknown.  That does not appear to be the case.  Listed below are some of the findings in Every Shot Counts:

·              Players putt better when they win tournaments than when they lose them-p. 24.

·              Tournaments are won by players excelling in different parts of the game-p. 17.

·              Bubba Watson gained strokes because of his driving- p. 91.

·              Luke Donald is a good putter and Sergio Garcia is not–p. 94.

·              Tiger’s secret weapon is approach shots-p. 116.

These same findings could be found by mining other statistics published by the PGA Tour.  Tiger Wood’s secret is also revealed by his greens in regulation statistic (GIR).  Woods was ranked first in GIR in both 2006 and 2007.  The PGA Tour also publishes a player’s GIR from various distances so a player can determine his relative strength at various approach shots.  Such detailed information is not available from Broadie’s strokes-gained statistic.

The contribution of the strokes-gained statistic is that it allows the total strokes-gained to be allocated to different shot categories.   A GIR ranking can show how good an iron player you are, but strokes-gained can estimate how much of your success is due to iron play.  A study of a player’s relative rankings can also give a clear picture of his strengths and weaknesses.  Strokes-gained, however, appears to be more definitive in determining why a player wins. This is what enraptured John Paul Newport when he wrote:

                Finely-focused strokes-gained analyses like these will make stat-watching more engaging in    the years ahead.[21]

Wide acceptance of strokes-gained, however, is not likely to happen for the following reasons:

1. Strokes-gained is an estimate based on a model of unknown accuracy.  Many of the assumptions in the strokes-gained model are clearly not true--i.e., distance is not the sole determinant of the difficulty of an approach shot or the value of a drive.  Nevertheless, Broadie takes his estimate of strokes-gained to the second decimal and never discusses the size of any possible error.

2. Alternatives to strokes-gained can be measured with precision and are easily understood.  GIR and average distance to the hole, for example, are not based on any underlying model.  They both can be measured with negligible error.  They are also statistics a television viewer can relate to his own game.  If two players are coming down the stretch in a tournament, and the on-course reporter says one player has gained 1.21 strokes by his approach shots and the other player has only gained 1.05 strokes, most viewers would be bewildered.   The reporter could go on to explain that the difference was due to the first player hitting shorter drives, but still hitting the greens with comparable accuracy.  That is, the first player had a lower strokes-gained for his drives.  It is unlikely the television director would encourage such descriptions of the action. 

3. It would be difficult to estimate strokes-gained in real time so its use would be restricted to post-mortems.  Even here the value of strokes-gained is questionable.  For example, Newport demonstrates that John Senden had the best strokes-gained short game performance of the season at the Valspar Championship—he hit three very close approach shots and holed out from 23 yards.  There does not appear to be much value in this statistic.  Can it be used to predict future performance?  Can it guide others to the path for victory on the PGA Tour?  The answer to both questions is “No.”  The statistic only demonstrated that to win on the Tour some players have to be good and lucky.

In summary, Every Shot Counts is not revolutionary.  Golf will not be transformed as of the book’s publication date.  As a statistic, strokes-gained has three failings: 1) the estimate of strokes-gained is based on a model of unproven validity which leads to measurement errors of unknown size, 2) it is not easily understood and possibly of little interest to the average player,  and 3) the statistic does not contain any information that is not present in other statistics kept by the PGA Tour.  These defects should deter the PGA Tour from expanding the use of the stroke- gained method to shots other than putts.      

 

 

.

---

[1] Dougharty, Laurence, “John Paul Newport: Not the Sharpest Wedge in the Bag I,” www.ongolfhandicaps.com, February 4, 1914.

[2] Broadie, Mark, Every Shot Counts, Gotham Books, New York, NY, 2014.

[3] Cochran, Alastair and Stobbs, John, The Search for the Perfect Swing, Golf society of Great Britain, London, 1968.

[4] Broadie, Mark, email to author, April 22, 2014.

[5] Another example of Broadie’s sophistry is his defense of strokes-gained putting.  Broadie writes “Before strokes gained putting stat, people used to count putts as the way of judging golfers (p. 33). Of course, serious people did not count putts but rather average putts per green in regulation.  In essence, Broadie built a straw man to convince the reader of the superiority of the strokes-gained method..  

[6] Cochran, op, cit, p.  201.

[7] Ibid, p. 200.

[8] Broadie, op.cit., p. 96.

[9]  Ibid, p. 96.

[10] Ibid, p. 116.

[11] Ibid, p. 28.

[12] The play of a hole can be described as a dynamic programming problem.  “Strokes-gained,” however, only gives the player a payoff value for every finishing position of his shot.  Golf is not the simple problem described by Broadie on p. 30.  The result of any shot is a stochastic process--i.e., there is a range of outcomes--not a deterministic one as in Broadie’s example.  To actually evaluate the millions of options for each shot would extend the playing time for a round into days.  Instead, a player uses simple algorithms to make his way around--i.e., hit it straight, don’t go for a sucker pin, hit away from trouble, etc.

[13] Broadie also takes quotes from Bellman’s autobiography (Eye of the Hurricane, World Scientific Press, Singapore, 1984) without attribution.  Since there are no footnotes in Every Shot Counts, this may have been an editorial decision.

[14] Broadie, op.cit. p. 83.

[15] Cochran, op. cit., p. 191.

[16] Ibid, p. 200.

[17] Newport, John Paul, “A New Golf Statistic Goes for a Test Drive,” Wall Street Journal, March 21, 2014.   In this column, Steve Evans of the PGA Tour identified the problem of a purely distanced-based system.  The same objection could be raised about the Strokes Gained-Putting statistic—i.e., not all 20 foot putts are of equal difficulty.

[18] Broadie, op. cit., p. 57.

[19] Ibid, p. 253.

[20] Lewis, Michael, Moneyball, W.W. Norton, N.Y., NY, 2003.

[21] Newport, op. cit.