The Francis Scheid Affair

            Nothing illustrates the bankrupt nature of the USGA handicap program better that what I term the “Francis Sheid Affair.”  It exposes the dearth of research behind handicap policy, and the unwillingness of the USGA to address criticism of the handicap section.

            The affair started innocently enough when I read a paper by Francis Sheid of the USGA, presented at the Second World Scientific Congress of Golf (1994).  The Congress guidelines required papers to be original.  The organizers of the Congress properly believed that if papers presented were merely rehashes of old work, the Congress would suffer in prestige. 

Since Sheid was a member of the USGA Handicap Research Team, I was expecting an article on the cutting edge of research on handicap issues.  Scheid’s article was entitled “The Search for The Perfect Handicap.”[1]  In this article, Sheid evaluated various handicap systems.  He concluded that “the current official handicap systems do about has well as any.”  It was an interesting piece, but clearly not of overwhelming significance.

            I was struck, however that I had seen the title somewhere before.  I went back to the proceedings of the First Scientific Congress of Golf(1990).  In a paper presented at this conference, Scheid had included the following reference:

 

"Scheid, F. J. (1978) The search for the perfect handicap.  Proceedings of the Winter Simulation Conference."

         The reference is not well documented.  There is, for example, no indication which organization published the paper.  This had the effect, intended or not, of deterring anyone from finding this paper.  However, with the help of some of the best research librarians in the United States the article was uncovered.[2]

            For all intents and purposes Scheid’s 1978 and 1994 articles were identical.  This meant 1) Sheid violated the conference guidelines calling for original research , 2) Sheid, a lead member of the Handicap Advisory Team, had not done anything in the period 1978-1994 he thought worthy of publishing.

            Scheid’s 1978 paper was sent to Martin Farrally, the Congress chairman, along with an analysis of the similarity between the two papers.  I thought he would be outraged that the USGA had pulled such a stunt on his conference.  Unfortunately, he was not.  Instead, Farrally replied:

“Since I have no competence in golf statistics I am not able to comment on whether Scheid’s paper was a regurgitation of 16 year old piece of work.  As the Congress director, I have to trust the judgement (sic) of the theme co-ordinators.”[3]

            Farrally did not say that he would investigate the claims.  Here merely stated that if it got by his theme coordinators, that was good enough for him.  He did not question his theme coordinator if he/she was aware the 1994 paper was a re-write of an earlier work.  Since some of the coordinators were paid consultants to the USGA, it is not likely they judged USGA generated papers too harshly.  Moreover, the entire Congress depended upon support from the USGA.  Apparently Farrally, felt that if the USGA wanted to slap the Congress in the face that was all right, because they had paid for that privilege.

            Even though Farrally refused an independent investigation of the affair, he did send my complaint on to Scheid.  Sheid wrote to Farrally that he was innocent.[4]  Farrally did not send a copy of this letter to me but did send a copy to David Fay, Executive Director of the USGA.  Fay mistakenly believed that Sheid presented a credible explanation and forwarded Scheid’s letter to me. 

Sheid’s defense was incredulous, and Fay should have spotted it.  In Fay’s defense, he knows little about statistics.  To his discredit, he never made any effort to settle the dispute, and became the leader in the effort to protect Scheid from the punishment he deserved.

            The weakness of Scheid’s defense can easily be seen.  His defense is shown below in italics.  A rebuttal of each of his points follows.

The original research (i.e., the 1978 paper) was based on the limited data available to me at the time.  I completely repeated all experiments using an entirely new and much larger data base.

 A comparison of the size of the samples used in each paper does not indicate a larger data base in Scheid-1994:

· Scheid-1978 – “At each of the courses for which data was in hand, fifty pairs of golfers (at eighteen courses) were chosen…At least…80,000 matches were played at each club to produce these results.

· Scheid-1994 – “at each of several clubs fifty pairs of golfer were select…About 80,000 matches were simulated a each club.

Since in common usage “several” is less than eighteen, it appears the sample size in 1994 was actually smaller than in 1978. Unfortunately, Scheid did not even meet the minimal scientific standard of documenting his sample size in his 1994 paper, so his claim could be conclusively disproved.

Scheid, however, proves his own guilt in another paper written for the 1994 Congress.[5] Sheid references two papers entitled The Search for the Perfect Handicap (1979), and the Search for the Perfect Handicap Part II, 1980. If he really did the research in the 90’s as he claims, why wouldn’t he reference the later work? The obvious answer is that there was no later work.

The results were reassuringly close to the original ones though not identical. Scheid tries to make the argument that since the results were different, there must be two different research studies.  His argument does not stand up to even a cursory inspection.

            Tables 1 and 2 present a comparison of the error tables contained in Scheid’s two papers.  Tables1 and 2 only report handicap types that are presented in both Scheid-1978 and Scheid-1994.  Not one handicap type reported in both of Scheid’s papers has a different error.  Where a comparison can be made, all the results are identical!

            The probability that the median error for fourteen different handicap types would remain exactly the same in two different samples is infinitesimal (the probability of this result is approximately .0002).

 Table 1

Median Errors, Match Play

 

Error	Handicap Type Scheid-1978	Handicap Type Sheid-1994
.4	Mean, (2-19), (3-18), (4-17), (8,13)	Mean, (2-19), (3-18), (4-17), (8,13)
.5	(1-15), MEDIAN,(6-15), (B15), (8)	(1-15), MEDIAN, (6,15), B15,(8)
.7	B12	B12
.8	USGA	USGA
1.0	(2), (1-5)	(2), (1-5)
>1.0	PT12	PT12

 

Table 2

Median Errors to Equalize Percentile 1 Scores

Error	Handicap Type Sheid-1978	Handicap Type Scheid-1994
1.2	(1), (1,5), Norm*	(1), (1-5),NORM
1.4	(2), (2-19), USGA, (1-15)	(2), (2-19), USGA, (1-15)
>1.4	PT12, MEDIAN,	PT12, MEDIAN

·         NORM is reported as 1.3 in Scheid-1978.  The error category “1.3” is not reported in Scheid-1994.  It is assumed that the error in NORM was rounded down to at “1.2” error.

 

After all, reproducibility is important in science and I was delighted with the confirmation that the larger experiment brought.

 

To get exactly the same results 16 years apart is indeed a spectacular finding.  But if it was so important, why did not Scheid report it in 1994?  The only reasonable conclusion for omitting such a spectacular finding is that Scheid did not want to draw attention to his previous paper.

 The earlier paper…was offered about twenty years ago.  I was anxious that the scientific golf community hear of its major result.  The Congress seemed the ideal place to do this.

 If Scheid was so anxious, it has to be asked why he did not submit his paper to the 1990 Congress rather than wait until 1994?  It is even more curious that his 1990 Congress submission (“On the Normality and Independence of Golf Scores”) contained a reference to Scheid-1978, but Scheid-1994 did not.  In 1990 paper he did not allude to his “major “ finding.  He simple listed the paper in the reference section even though it had nothing to do with the body of work.  Simply put, the arguments Scheid makes in his defense are not consistent with the facts in the case.  

The paper also included a new result, that even a two-dimensional system based on the normal model would not bring a significant improvement in accuracy… 

This is not a new result but appears in Scheid-1978.  In Scheid-1978, he writes “The normal model also does as well as any…(But) official USGA and British handicaps take a close second place.”  Scheid-1994 draws the same conclusion

Scheid does add an analysis of team play in Scheid-1994.  This is a minor addition, however, and has no effect on the overall conclusion of the research.

            The USGA had steadfastly refused to admit that Scheid violated the rules of the Congress.  David Fay Executive Director of the USGA, wrote: “we will not censure Francis Scheid or offer an apology.”[6]  Never, however, has the USGA offered an affirmative defense of Scheid’s conduct.  This has been consistent with the USGA policy of dealing with criticism – attack the messenger and ignore the message.

            The concern here is not that Scheid submitted old research, but that his sixteen-year old work was the best the USGA had to offer.  The USGA should be the recognized authority on handicapping as it is on equipment testing.  The USGA has not shown the breadth and depth of research on handicapping issues to deserve such status.

            During the past ten years, major changes in the handicap system have been implemented – the Slope System, a revised and re-revised equitable stroke control, and the reduction in index for exceptional tournament performance.  Yet the USGA has not published any empirical research demonstrating these revisions have had a positive effect on the equity of competition. 

            The poor performance of the USGA is due in part to its monopoly status.  Members of the handicap research team appear to be selected because they had some commercial tie to the USGA or its executives.  There is no record of the USGA seeking to hire the best talent in the country to study handicapping problems.  Instead the USGA has relied on the “old crony” system and the results show it.  As documented throughout blog, the USGA has made numerous mistakes and questionable judgment calls on handicap policy.  Unfortunately, there are no forces on the horizon that could change the USGA.  The USGA Executive Committee will continue to rule by its inherent motto, “We’re God, and you’re not.”

 

---

[1] Sheid, F.J., “The search for the perfect handicap,” Science and Golf II: Proceedings of the World Scientific Congress of Golf,  Edited by A.J. Cochran and M.R. Farrally, E & FN Spn, London, 1994.

[2] Scheid, Francis, the Search for the perfect handicap, 1978 Winter Simulation conference, IEEE, 1978

 [3] Letter from Martin Farrally, Director of the World Scientific Congress of Golf to the author, December 13, 1994

[4] Letter from Francis Scheid to Martin Farrally, April 13, 1995.

[5] D.L. Knuth, F.J. Scheid, and F.P. Engel, “:Outlier identification procedure for reduction of handicaps,” Scheid can Golf II: Proceedings of the 1994 World Scientific Congress of Golf, E & FN Spon, London, 1994

[6] Letter from David Fay, Executive Director of the USGA to the author, December 7, 1998.

The USGA - The Dishonarable Company of Golfers

Introduction - The history of golf is replete with stories of golfers calling rulings on themselves.  Bobby Jones is noted for calling penalty strokes on himself for violations that no one else saw.  One such call may even have cost him the 1925 U.S. Open.  Indeed it is the honor in the game that most distinguishes golf from its competitors on Sunday television.  In football and baseball, it is all right to claim you caught the ball even if you didn’t.  Lying is an accepted if not an integral part of those games.

            Given the history and tenets of golf, one would expect the United States Golf Association (USGA) to exemplify all of the virtuous attributes ascribed to the game.  That certainly was my expectation, but it was dashed on the rocks of reality by the USGA’s reaction to my paper questioning the value of the Slope System (see Is the Slope System Worth the Effort - posted 9/20/2012).  The USGA engaged in lying, fraud, and character assassination in an attempt to bury my work.  In the USGA’s case, the bureaucratic instincts of survival clearly overwhelmed any inclination to do the honorable thing.

 In the Beginning - My research on the Slope System was born out of curiosity.  The Slope System was hard to use and difficult to understand.  To be implemented it had to have significant benefits.  I wrote to the USGA asking for studies of the Slope System that demonstrated its efficacy.  In response, the USGA only sent a description of the Slope System. 

Having observed bureaucratic behavior for over ten years as a policy analyst at the Rand Corporation, I found the USGA’s failure to send any studies probably meant there were not any.  After all, if there were strong scientific support for the Slope System there should have been no hesitancy to release it.

            Given the USGA’s reluctance, I went ahead with two tests of the Slope System reported in Is the Slope System Worth the Effort- posted 9/20/2012.  The two tests were not meant to be definitive, but to 1) demonstrate a methodology for testing the Slope System and 2) determine if more research needed to be done. The research concluded, qualified by the limited sample size, that the Slope System was a second order improvement in the handicap system at best. 

            The report “The Slope System: A Review and Evaluation” was submitted to the USGA for its review.  The USGA’s response at this was encouraging.  Francis Scheid of the USGA’s Handicap Research Team wrote:[1]

It is always a pleasure to find that someone else is involved in handicap research and your work shows the competence that the subject deserves…In summary, you have obviously made an important contribution.  I hope you can avoid the temptation to go for the front page of the Washington Post and suggest that instead we continue to refine what is basically a sound idea.

            This was in essence a pat on the head.  I might have accepted it if Scheid had not gone on with some rather silly reasoning in defense of the Slope System:

The reality of the slope concept should not be doubted.  Common sense points directly to it… (If) A outbowls B by 20 pins a string on average, and B bests C by 30, then Las Vegas will expect this field of three to show gaps of 60 and 90 strings over a three string event.  (This example)…illustrates a general principle, that differences in ability are magnified by increasing the level of difficulty of the assignment…”

 

            In response to Mr. Scheid, I had to point out his example did not support the concept of a Slope System.[2]   To be kind, I said nothing about his choice of a bowling analogy nor did I take it as an insult.  I did say:

Las Vegas is simply assuming an independence among trials (i.e., strings), and computing the expected value of three strings to be three times the expected valued of one string.  Your implied logic that if this bowling example is true, then the Slope System is valid is simply false. The Slope System tried to deal with how to standardize a trial (e.g., what should your bowling handicap be if the lane was lengthened by 20 feet?), rather than cumulative differences over a number of trials

 

            Apparently any criticism was too much for Mr. Scheid.  He characterized my effort as hate mail[3] even though my letter concluded with:

I have the greatest respect for the USGA and for your work for the past twenty years in studies of handicapping.  For the love of the game you dedicated your talent to making improvements to ensure more equitable competition.  I hope you can believe my relatively modest effort was undertaken with the same goal in mind.

Not exactly my definition of hate mail type material, but perhaps I have not led the cloistered existence of Mr. Scheid.

            But that was not enough for the USGA.  Dean Knuth, the Director of Handicapping for the USGA, had to pile on: [4]

“I share Dr. Scheid’s repulsion of your style of correspondence and also question who would even be interested enough in your handicapping work to give you the ‘economic advantage’ that you admit to seek.”

Of course I had not sought and economic advantage – though that term really has no meaning – since there is no money to be made in research on golf handicapping.  Knuth tried to discredit my research by impugning my integrity. It was something he would repeat on several occasions in the future. 

While ideas certainly should have been debated on a level playing field, the USGA chose instead to shovel dirt on a minor opponent.  And remember these are the men who so often remind us that they are guardians of the integrity of the game.

The World Scientific Congress of Golf - I had also sent my research paper to Dr.

Alastair Cochran, author of The Search for the Perfect Swing.[5]  He wrote back:

If repeated with a similar outcome elsewhere, the results of your study of play from two different sets of tees certainly would cast doubt on the need for the Slope System….You may wish to consider submitting one or more of your papers to the Second World Scientific Congress of Golf, to be held in St. Andrews in July 1994.  I have taken the liberty of putting your name on the list to receive advance notice of this.

Since Cochran was one of the leaders of applying science to the study of golf, I was encouraged by his generally positive response.  I was particularly excited by the opportunity to share my research at the Congress.

            The paper was submitted to the Congress for review in November of 1993.  The Conference Director, Martin Farrally, informed me that all decisions would be made by February 15, 1994.  I also learned at this time that the USGA was a co-sponsor of the Congress.  While I found this troubling, I did not believe the USGA would be so blatant as to use their power have my paper excluded from the Congress.  I was wrong.

            The paper was rejected by the Congress, but the reviewer who rejected the paper demonstrated both his bias and ignorance of the subject area.  The reviewer carped about the small sample size.  This argument was not compelling for the following reasons:

1.   The Congress Has Published Papers With Far Less Data and Smaller Sample Sizes - At the First Congress, Stroud and Riccio, members of the USGA Handicap Procedures Committee, published a paper entitled "Mathematical underpinnings of the slope handicap system."  While they described the system and extolled its benefits, they did not present one shred of empirical evidence that the system would increase the equity of competition.  Moreover they were allowed to cite articles that are not available to the general public, but only to select members of the USGA community.  This is hardly consistent with the principles of scientific inquiry that the Congress should represent.

            Dean Knuth, Director of Handicapping of the USGA, had a paper entitled "A two-parameter golf course rating system" presented at the First Congress.  Again there is no research on the benefits of the Slope system, but merely a description of how ratings are done.  For the course rating system, Knuth only used data from seven U.S. Amateur Championship courses in his sample.  He then threw out 41 percent of the hole scores sampled without providing a sound reason.  The estimates of the coefficient of the bogey rating model were based on the scores of only 60 players!  If such a skimpy sample is of sufficient size on which to base the entire USGA Handicap System, then my sample size should be adequate to question the efficacy of that system.

                  To criticize my paper because of its limited sample size is to hold it to a higher standard than research submitted by the USGA.

 

2.   The Sample Size is Not that Small -  The inter-club test of the Slope System involved most courses in Southern California.  This is probably the largest such team competition held anywhere.  And any statistical problems certainly do not stem from sample size.

            My study concluded that tough courses (i.e., high slope ratings) did not have the big advantage that the Riccio and Stroud paper postulated.  Riccio and Stroud only gave theoretical arguments, of course, and had no empirical evidence on which to base their claim.  My results were based on over 100 team matches in each year of the study.  Therefore, a reasonable person would have to conclude that it is not sample size that is the deciding factor in the selection of papers.

 

3.    The Value of the Paper is in Presenting a Methodology that Can be Replicated by Others to Test the Merits of the Slope System -  My paper presents a methodology for testing the slope and course ratings among various sets of tees at a course.  I believe this is important for golf associations in validating the accuracy of their ratings.  I have demonstrated this methodology for only one course.  While my result raises questions about the Slope System, it cannot serve as a general indictment as I readily admit.  With my methodology, however, golf associations could test the slope ratings at thousands of courses.  By rejecting my paper, however, the associations are deprived of the methodology to perform such tests.

             The reviewer also claimed that I had erred since the Slope System was introduced in Southern California in 1990 and not 1991 as I reported in my paper.  I gave Farrally the phone number of the Southern California Golf Association (SCGA) so he could check who was right.  Farrally never made the call.  Since the SCGA had both supplied the data for the study and reviewed the results no such error could have occurred.

            If Farrally was embarrassed by his reviewer’s mistake, he did not show it.  Instead he went on to claim independence from the USGA:[6]

“May I assure you there is no politics involved in the rejection of your works as a paper.  We make no secret that the R & A and the USGA are major sponsors but the congress is run by a university with full academic freedom.” 

 

            Farrally’s claim of independence rings hollow.  Without the USGA and R & A sponsorship, the Congress and Farrally’s job as director would not exist.  Moreover , the USGA is not without an institutional interest.  The USGA also has an economic interest in maintaining its omnipotence on all handicapping matters.  The USGA sells handicap services much like Ping or Cobra sells golf clubs.  Therefore, the Congress conference has the appearance, if not the reality, of a conflict of interest in the area of handicapping. But let’s examine the evidence.

            Every paper submitted by the USGA was accepted.  These papers were hardly earthshaking.  L.J. Riccio of the USGA Handicap Procedures Committee did a study of Tom Watson’s play at the US. Open – The Ageing of a Great Player: Tom Watson’s Play in the U.S. Open from 1980 to 1993.  A longitudinal study of one player cannot say anything about the effects of ageing for the population as a whole.  Since the venue and weather change each year, the study is of even limited value in assessing Tom Watson’s performance over time.  Therefore, the criticism of my paper because of its limited sample size, was simply a ruse to the USGA bidding.

Dean Knuth submitted a paper that was just a description of the Adjustment for Exceptional Tournament Performance.  Knuth presented no evidence that the new procedure had any effect on the equity of competition.  Though I was criticized for not having sufficient empirical findings, Knuth paper was embraced even though it had no findings.

Finally, Francis Sheid of the USGA submitted a paper, The Search for the Perfect Handicap, that by its title was not the definitive research that was demanded of my paper.  It was to turn out that Scheid’s paper was just a rehash of a paper he had written 16 years earlier.  A good reviewer would have refused Scheid’s paper because it was not original.  Since USGA paper’s probably got minimal scrutiny at best, it was accepted.

In taking the USGA’s money, the Congress received a batch of mediocre papers of questionable scientific merit and little policy value.  The Congress made a bad bargain that exposed that it is a creature of politics and not science.[7]

            Farrally tried to defend the review process:

“Your paper was reviewed by two separate academics, both of them well respected statisticians working in Universities, on British and one American.  Because of the sensitivity of this issue I am prepared to inform you that the American reviewer was not Frank Scheid.  I can also confirm that having spoken to both reviewers, I have been assured that the content of neither review was disclosed to anybody else but myself.”

 

            Farrally’s defense strongly implies the reviewers were independent of the USGA.  I learned, however, that my paper was reviewed by Burt Lieberman who is not a statistician but an associate professor of mathematics at Polytechnic University – a commuter school in Brooklyn, New York.  Mr. Lieberman has no publications in either econometrics, quasi-experimental design, or the measurement of human performance in golf.  He has, however, been a consultant to the USGA since 1979!  I guess this little fact slipped Farrally’s mind.

            I further learned that the review of my paper had been discussed at length with members of the USGA Handicap section before being sent to Farrally.  This is easily proved by a letter Dean Knuth sent to the head of the Connecticut State Golf Association:[8]

“An independent Congress of Golf review committee composed of professors from Great Britain and the United States ‘peer reviewed” his (Dougharty’s) report and found it unfit for their proceedings.”

 

            Now how would Knuth know of the reviews unless he had contact with the reviewers?  Farrally continued to maintain that “neither of the review were seen by anyone else except myself and Alastair Cochran.[9]”  Did Farrally take the most direct step and ask Knuth how he knew of the results?  Of course not.  This would only embarrass the USGA and was something Farrally could not afford to do.  Farrally continued to maintain the USGA had nothing to do with the review even against overwhelming evidence to the contrary.  Farrally was certainly a good soldier in the cause of the USGA.  It is not surprising he continues as Congress Director and the USGA continued its financial support.

            The response of David Fay, Executive Director of the USGA, was equally disappointing.  After Fay was sent all of the supporting evidence he wrote:[10]

”In our judgement (sic), we concluded that yet another reply to you would accomplish nothing.  We know your feelings; we happen to disagree.”

 

Though Fay must have been aware of the USGA’s involvement, he took no corrective steps nor did he offer an apology for the actions taken by his organization.  If the rejection of my paper was not embarrassing enough for the USGA, the acceptance of Francis Sheid’s was even more so.  That sordid affair is detailed in the next post, The Francis Scheid Affair.

---

[1] Letter to the author from Francis Sheid, February 20, 1991.

[2] Letter from the author to Francis Scheid, February 27, 1991.

[3] Letter to the author From Francis Scheid, March 18, 1991.

[4] Letter to the author from Dean Knuth, April 18, 1991.

[5] Cochran, Alastair and Stobbs, John, The Search for the Perfect Swing, The Bootlegger (reprinted), Grass Valley, CA 1989.

[6] Letter to the author from Martin Farrally, February 4, 1994.

[7]  The Congress showed a similar bias in a paper submitted by a writer from one of the sponsoring organizations, Golf Digest (Australia).  The paper contained no substantive research, and was generally a lament about the poor television coverage of Australian tournaments.  The writer concluded his paper with, “As journalists continue to be so badly paid, golf writers will continue to look for other public relations work to supplement their paltry incomes.”  I assume this writer’s paltry income was supplemented by a free trip to the St. Andrews, the Congress site.  Others used the Congress as a tax write-off.  U.S. taxpayers were picking up part of the tab for many Congress participants to play the Old Course.

[8] Letter from Dean Knuth, USGA Director of Handicapping to Russell Palmer, Executive Director of the Connecticut State Golf Association, August 25, 1994.

[9] Letter to the author from Martin Farrally, November 22, 1994.

[10] Letter to the author from David Fay, December 7, 1998.