In a recent post,[1]
it was argued that many of the findings in Mark Broadie’s book, Every Shot Counts,[2]
were previously presented in a book written over 40 years ago by Alastair
Cochran.[3] Professor Broadie challenged that conclusion
and made other criticisms of Cochran’s work.[4] The purpose of this note is to examine three
questions: 1) Are Broadie’s criticisms
of Cochran’s work valid? 2) What are the
true origins of the strokes-gained concept, and 3) Is the strokes-gained
statistic either revolutionary or of value?
Each question is discussed in turn.
1. Are Broadie’s
Criticisms of Cochran’s work valid? – Broadie’s criticisms are in italics
followed by an analysis of their validity:
Even
though Cochran and Stobbs title their chapter “Long Approach Shots-Where
Tournaments are Won,” they actually present no evidence that this is the case.
It is true Cochran does not write that long iron play is the
ultimate determinant of who wins. That
would be foolish, and Cochran is clearly not a fool. Players win for a variety of reasons as
Broadie’s own work shows. Cochran merely
argues iron play is important and
presents a table showing the difference in iron play between the top nine and
bottom nine players (see Table 31:6 below).
Normalizing for the play of non-iron shots, Cochran concluded the top
nine players had a five stroke advantage over the bottom nine players. This is clearly evidence that iron play is
important in winning. To claim that Cochran presents no evidence of the importance
of iron play is clearly wrong. Perhaps
Cochran can be criticized for an overly definitive chapter title. But you cannot argue Cochran did not identify
the importance of long iron play long before the publication of Every shot Counts.
Table 31:6 Long Approaches at Birkdale: How the leaders compared with
the tail-enders
|
|
Distance from which shot is played (Yards)
|
||||
|
140-160
|
160-180
|
180-200
|
200-220
|
||
|
Median finishing distance from hole (yards)
|
Top nine players
|
9.8
(46)
|
9.0
(22)
|
13.6
(63)
|
14.5
(50)
|
|
Bottom nine players
|
12.0
42
|
10.8
(17)
|
13.8
(64)
|
17.7
(54)
|
|
Broadie saves his most strident criticism for Table 31:8 that
is reproduced below:
This is the main table in the chapter,
on which the title is based. It (Table 31:8) compares a 50% reduction in
long approach shot errors (completely unrealistic given the 14% above) with
“doubling the accuracy of putting” (also completely unrealistic) with hitting
drives 20 yards further and having them all finish in the fairway (also completely
unrealistic). There are two problems with this. First, the “strokes
gained per round” from these assumptions is not explained and I’m pretty sure
is not correct. Second, even if we accept those numbers, it makes no
sense to conclude anything from hypothetical, unrealistic assumptions that are
not at all comparable. It would be like saying the strokes gained
per round from hitting drives 3 yards further is much less than one-putting
twice as often, therefore putting is “most important.”
Cochran and Stobbs (see below) write “we are not
implying these improvements are equally easy to achieve” yet they still base
their conclusions (“But it would not be too far to conclude”) solely on this
one table with completely arbitrary assumptions. It is crucial to base conclusions on comparable
improvements; completely arbitrary assumptions lead to conclusions that are
completely arbitrary.
Table 31:8 How much the pros at Birkdale would have gained by drastic
improvement in their game
|
|
Gain in Strokes per Round
|
|
By “doubling” accuracy of putting
|
4.2
|
|
By “doubling the accuracy of short approaches
|
1.7
|
|
By “doubling the accuracy of long approaches
|
5.5
|
|
By 100% accuracy and extra 20 yards on drives
|
2.2
|
|
Total
|
13.6
|
Broadie
engages in sophistry in attempting to prove his points.[5] Nowhere does Cochran maintain this table
proves the primacy of iron play. In
fact, this table is never cited in the
text. This would be strange if Cochran
thought this to be his “main table.” Cochran
does conclude “it would not be going too far to conclude that it is in full
iron play as well as putting that the main difference in caliber makes itself
felt between different levels of top class professional golfers and the degree
of success they have in tournaments.” [6] His conclusion is based (I assume) on the
study of difference in putting shown in Table 29:6 (Putting at Birkdale: How
the leaders and the tail-enders compared with the field as a whole) and Table
31:6 shown above. To argue that Cochran
based his conclusion on Table 31:8 is a misstatement of the facts.
Cochran readily admits the improvements
assumed in Table 31.8 are unrealistic.[7] Broadie complains that not
much can be drawn when unrealistic assumptions are made. Yet that does not stop
Broadie from analyzing the impact of hitting the ball 20 yards further.[8] He argues this assumption is unrealistic but
it is important to understand “the trade-off between distance and accuracy for
course strategy.”[9] Cochran was merely using the same literary
device as Broadie and is undeserving of criticism.
The gain in strokes per round shown Table 31:8 could
represent estimates of the marginal value of improvement in each category. The marginal value of improvement must be
weighed against the marginal cost of that improvement to arrive at the optimum
practice plan. For the average player,
Cochran concludes improving his iron play is probably the best use of his
time. This is not much different than
Broadie’s finding “The biggest contributor to scoring? Approach shots, which contribute 40% to the
total strokes gained.”[10]
Broadie complains the methodology behind the “strokes-gained”
calculation is not explained. Cochran
did show estimates for iron play in Table 31:7 shown below.
Table 31:7 - The
benefits of improved iron approaches: an estimate of strokes gained by halving
their inaccuracy (Abridged)
|
Hole
|
Length of hole remaining after
250 yard drive
|
Shots
to get down from this distance
|
|
|
Normal
Standard
|
Improved
Standard
|
||
|
1
|
243
|
No long approach required
|
|
|
2
|
177
|
3.13
|
2.74
|
|
9
|
160
|
3.05
|
2.71
|
|
18
|
200
|
3.23
|
2.79
|
|
Total
|
43.84
|
38.39
|
|
To examine the benefits of improved iron approaches Cochran
compares the hole scores for players with standard accuracy with a player who
is 50 percent more accurate. Broadie is
correct that Cochran does not detail the method used to estimate the “shots to
get down.” (I assume this was the book editor’s decision much as in Broadie’s
book where any description of his simulation model is omitted.) It is possible to speculate on possible
methods and show that Cochran’s estimates are not too far off, if at all. One possible method would be to first
estimate the normal standard finishing distance from the hole. Cochran has established that the medium
finishing position of an iron shot is approximately 7.5% of the starting
distance. At hole 9, for example, the
average finishing position would be 12 yards.
From the graph in Fig. 29.2, the average number of putts from 12 yards
is 2.05. Therefore the standard number
of shots to get down from 160 yards is 3.05 (2.05 +1).
If a player improved his accuracy, his finishing distance
would be 6 yards. The number of putts to
get down from this distance is approximately 1.82. With improved iron play, the player would
get down in 2.82 strokes. Cochran,
however, reports the player would get down in 2.71 strokes. Cochran could have used a more sophisticated
method where a player with an improved standard hit more greens and/or missed
more bunkers. In any case, Cochran’s
estimates in Table 31:7 appear to be in the ballpark. Contrary to Broadie’s assertion, Cochran does
not base a conclusion on them.
In summary, Broadie’s criticisms are not valid. Cochran could have presented more detail
about his work, but there does not appear to be any evidence of poor
scholarship as Broadie asserts.
2. What are the True
Origins of the Strokes-Gained statistic?”
- In Every Stroke Counts, Professor
Broadie takes credit for the “strokes-gained” concept of measuring player
performance:
Strokes-gained
is the name for this new way of measuring shot quality. It uses the same unit-strokes – to calculate
the skill of the many kinds of shots taken throughout each round of golf. The origins of most new ideas can be traced
to the earlier work of others, and strokes-gained is no exception. The term owes its heritage to a brilliant
applied mathematician of the mid 20th century (Richard Bellman), and
a grand theory he called “dynamic programming” developed at the dawn of the
computer era….Using this technique, I (emphasis
added) developed a way to compare golf shots and quantify a golfer’ skill.[11]
I do not believe the lineage of “strokes-gained” goes back
to the works of Richard Bellman.[12] The reference to Bellman seems soley intended
to give “strokes-gained” some mathematical credibility by association.[13] Stripped down to its bare essentials,
Broadie had access to a big pile of data from which he calculated the average
score to complete a hole as a function of distance. You take the average score from Point A and
subtract the average score from Point B.
The difference minus one is the strokes-gained for the shot. This is not dynamic programming as Broadie
implies, but rather an exercise in second grade arithmetic. Broadie admits as much later in the book when
he writes “Once you have access to the data base showing average strokes to
hole out from any given distance, there’s no rocket science involved in calculating
strokes-gained, just subtraction.”[14]
I strongly believe Broadie owes an intellectual debt to
Cochran. Both Broadie and Cochran tried
to identify the importance of each type of shot is isolation from the
others. Only Cochran did it first. Broadie refined the methodology to make it
player-specific while Cochran’s results were only tournament-specific. A review of how each researcher found the
strokes-gained will reveal the similarities and differences in their
approaches.
Putting – Broadie
has estimated the average number of putting strokes it takes to hole the ball
for various distances from the hole. The
average number of strokes minus the number of actual strokes taken is the
putting strokes gained or lost on
that hole. Cochran’s method is
similar. Due to data limitations he
compares the top nine players with the bottom nine players in an attempt to
explain the difference in performance.
Cochran makes some calculations about the distribution of the lengths of
first putts “assuming that their play up to the green has been the same—which,
of course, it wasn’t.”[15]
Cochran then applies the probabilities of each group to make a putt of a
certain length. He found the top nine
players gained about a tenth of a stroke per hole over the bottom nine players
due to putting. Cochran attempted to
normalize the distance of the putt in order to isolate the value of putting. This is the basic idea behind Broadie’s strokes-gained
putting.
Long Approach Shots
– Broadie has estimated the average number of shots it takes to hole a ball
from various distances and positions.
The strokes gained from a long approach shot is calculated as follows:
Strokes- Gained = Average Number of Strokes to Hole from Starting
Position – (Average Number of Strokes
to Hole from finishing position +1)
As discussed under the previous question, Cochran employs
much the same method. Cochran writes:
“Taking
all eighteen of the players concerned to have hit the average drive of the
whole field, and to have played their
short approaches and putted to the
average standard too, the team were able to show that the difference in
standard of strokes from 140 to 220 yards gave the top nine an advantage of
about one and a quarter strokes per round over the bottom.”[16]
ShotLink allows Broadie to be more precise. He can take a player’s actual drive and long
approach and estimate the strokes gained.
This allows Broadie to estimate the shots gained for each player. Without the vast data of ShotLink, Cochran
had to assume all players started from the average drive. He then applied the long approach accuracy
estimates of Table 31:6 for each group of players to determine where the long
approach shot finished. From the
finishing positions the number of strokes needed to hole the ball was
estimated. The difference in the
estimates of the top nine and bottom nine players was the estimate of the
strokes gained by the better players.
Broadie’s and Cochran’s methods are remarkably similar in concept.
Driving - Broadie has used ShotLink data to estimate
the shots needed to complete a hole from any distance and a variety of
positions (fairway, rough, sand bunker).
He takes the estimate of the number of shots to complete the hole from
the tee and subtracts the number of strokes to complete the hole from where the
drive finishes plus one. While Broadie
never presents any evidence of the accuracy or inherent variability of these
estimates, they do allow for a straightforward calculation of strokes-gained
driving for each player. Cochran was
hampered by his small data set. The difference
in driving between the top nine and bottom nine players was only 7 yards. Cochran, by taking into account calculations
made for the other types of strokes, showed the top nine players gained a half
a stroke per round by their driving over the bottom nine. Cochran does not specify his
calculations. It can be assumed that he used
the same method as for the other types of shot.
That is, the longer drives led to more accurate long approaches, which
led to fewer putts and a lower average score.
Both researchers reached the same conclusion that when it comes to
driving, i.e., length counted for more than accuracy.
Short Approaches –
Both Broadie and Cochran have difficulty dealing with this shot. Under Broadie’s method, the number of shots
to finish a hole is a function of distance.
There may be so much variability in 20 foot chip shots, for example,
that Broadie’s method needs more refinement before it can be implemented on the
PGA Tour.[17] Cochran found no significant difference in
the short approaches of the top nine and bottom nine players. Both groups had similar median finishing
distances for their short approach shots so no strokes were gained.
Broadie does not give Cochran credit for coining the term
“strokes-gained.” Instead that honor
goes to unnamed colleagues at MIT.[18] His only mention of Cochran comes in the acknowledgments
where he writes:
When
I started research into golf, I didn’t even remember that in high school I had
picked up from the library the now classic book Search for the Perfect Swing.
The authors Cochran and Stobbs in the 1960s were the first ones to
record and analyze individual shots.[19]
(That must have been one great high school library since it
acquired the book some twelve years before it was published in the United
States.) Broadie does not give Cochran
any credit for birthing the strokes-gained method. Given the similarities in their research
approach described above, Cochran deserves much more recognition than he is
given.
3) Is the strokes-gained statistic either
revolutionary or have value? - The
subtitle of Broadie’s book is “Using the Revolutionary Strokes Gained Approach
to Improve Your Golf Performance.” A new
statistic can be termed revolutionary if it changes the way the game is played
like Sabermetrics did for baseball.[20] Broadie presents no evidence that knowledge
of this statistic has altered a player’s decision-making the way Sabermetrics changed managerial decisions
in baseball.
The strokes-gained statistic could also be revolutionary if
it produced information about player performance that was previously
unknown. That does not appear to be the
case. Listed below are some of the findings
in Every Shot Counts:
· Players putt better when they win tournaments
than when they lose them-p. 24.
· Tournaments are won by players excelling in different
parts of the game-p. 17.
· Bubba Watson gained strokes because of his
driving- p. 91.
· Luke Donald is a good putter and Sergio Garcia
is not–p. 94.
· Tiger’s secret weapon is approach shots-p. 116.
These same findings could be found by mining other
statistics published by the PGA Tour. Tiger
Wood’s secret is also revealed by his greens in regulation statistic (GIR). Woods was ranked first in GIR in both 2006
and 2007. The PGA Tour also publishes a
player’s GIR from various distances so a player can determine his relative
strength at various approach shots. Such
detailed information is not available from Broadie’s strokes-gained statistic.
The contribution of the strokes-gained statistic is that it allows
the total strokes-gained to be allocated to different shot categories. A GIR
ranking can show how good an iron player you are, but strokes-gained can
estimate how much of your success is due to iron play. A study of a player’s relative rankings can
also give a clear picture of his strengths and weaknesses. Strokes-gained, however, appears to be more
definitive in determining why a player wins. This is what enraptured John Paul
Newport when he wrote:
Finely-focused strokes-gained analyses like these will make stat-watching
more engaging in the years ahead.[21]
Wide acceptance of strokes-gained, however, is not likely to
happen for the following reasons:
1. Strokes-gained is an estimate
based on a model of unknown accuracy. Many
of the assumptions in the strokes-gained model are clearly not true--i.e.,
distance is not the sole determinant of the difficulty of an approach shot or
the value of a drive. Nevertheless,
Broadie takes his estimate of strokes-gained to the second decimal and never
discusses the size of any possible error.
2. Alternatives to strokes-gained
can be measured with precision and are easily understood. GIR and average distance to the hole, for
example, are not based on any underlying model.
They both can be measured with negligible error. They are also statistics a television viewer
can relate to his own game. If two
players are coming down the stretch in a tournament, and the on-course reporter
says one player has gained 1.21 strokes by his approach shots and the other
player has only gained 1.05 strokes, most viewers would be bewildered. The
reporter could go on to explain that the difference was due to the first player
hitting shorter drives, but still hitting the greens with comparable
accuracy. That is, the first player had
a lower strokes-gained for his drives. It
is unlikely the television director would encourage such descriptions of the
action.
3. It would be difficult to
estimate strokes-gained in real time so its use would be restricted to
post-mortems. Even here the value of
strokes-gained is questionable. For
example, Newport demonstrates that John Senden had the best strokes-gained
short game performance of the season at the Valspar Championship—he hit three
very close approach shots and holed out from 23 yards. There does not appear to be much value in this
statistic. Can it be used to predict
future performance? Can it guide others
to the path for victory on the PGA Tour?
The answer to both questions is “No.”
The statistic only demonstrated that to win on the Tour some players
have to be good and lucky.
In summary, Every Shot
Counts is not revolutionary. Golf
will not be transformed as of the book’s publication date. As a statistic, strokes-gained has three failings:
1) the estimate of strokes-gained is based on a model of unproven validity which
leads to measurement errors of unknown size, 2) it is not easily understood and
possibly of little interest to the average player, and 3) the statistic does not contain any
information that is not present in other statistics kept by the PGA Tour. These defects should deter the PGA Tour from expanding
the use of the stroke- gained method to shots other than putts.
.
[1]
Dougharty, Laurence, “John Paul Newport: Not the Sharpest Wedge in the Bag I,” www.ongolfhandicaps.com, February 4,
1914.
[2]
Broadie, Mark, Every Shot Counts,
Gotham Books, New York, NY, 2014.
[3]
Cochran, Alastair and Stobbs, John, The
Search for the Perfect Swing, Golf society of Great Britain, London, 1968.
[4]
Broadie, Mark, email to author, April 22, 2014.
[5]
Another example of Broadie’s sophistry is his defense of strokes-gained
putting. Broadie writes “Before strokes
gained putting stat, people used to count putts as the way of judging golfers
(p. 33). Of course, serious people did not count putts but rather average putts
per green in regulation. In essence,
Broadie built a straw man to convince the reader of the superiority of the
strokes-gained method..
[6]
Cochran, op, cit, p. 201.
[7] Ibid, p. 200.
[8]
Broadie, op.cit., p. 96.
[9] Ibid,
p. 96.
[10]
Ibid, p. 116.
[11]
Ibid, p. 28.
[12]
The play of a hole can be described as a dynamic programming problem. “Strokes-gained,” however, only gives the player
a payoff value for every finishing position of his shot. Golf is not the simple problem described by
Broadie on p. 30. The result of any shot
is a stochastic process--i.e., there is a range of outcomes--not a
deterministic one as in Broadie’s example.
To actually evaluate the millions of options for each shot would extend
the playing time for a round into days.
Instead, a player uses simple algorithms to make his way around--i.e.,
hit it straight, don’t go for a sucker pin, hit away from trouble, etc.
[13]
Broadie also takes quotes from Bellman’s autobiography (Eye of the Hurricane, World Scientific Press, Singapore, 1984)
without attribution. Since there are no
footnotes in Every Shot Counts, this
may have been an editorial decision.
[14] Broadie, op.cit. p. 83.
[15]
Cochran, op. cit., p. 191.
[16] Ibid, p. 200.
[17]
Newport, John Paul, “A New Golf Statistic Goes for a Test Drive,” Wall Street Journal, March 21, 2014. In this column, Steve Evans of the PGA Tour
identified the problem of a purely distanced-based system. The same objection could be raised about the
Strokes Gained-Putting statistic—i.e., not all 20 foot putts are of equal
difficulty.
[18]
Broadie, op. cit., p. 57.
[19] Ibid, p. 253.
[20]
Lewis, Michael, Moneyball, W.W.
Norton, N.Y., NY, 2003.
[21]
Newport, op. cit.
