Zero Sum
How wonderful that we have met with a paradox. Now we have some hope of making progress. — Neils Bohr
Click on the section headings below to reveal/hide the contents
Substance: WordPress Style: Rachel
Posted Jul 25th, 2011 by ravi    / Permalink /

David DeGusta and Jason Lewis, writing in the New Scientist, raise the question “Is bias inevitable in science?“, and answer:

Stephen Jay Gould claimed unconscious bias could affect even seemingly objective scientific measurements. Not so.

Why not? Because “[scientific] method is so robust” that it can overcome the bias of scientists. Scientific Method has been studied greatly in the fields of History and Philosophy of Science – and cautiously defined and defended, criticised, found wanting or polemically dismissed - but DeGusta and Lewis are, in their piece, not concerned with these arguments, but rather with the particular story of “Gould’s skulls” and what that story tells us about scientific bias.

The backstory in short: In his analysis of the cranial measurements conducted by a 19th century scientist named Samuel Morton, Gould found that Morton had “manipulated his samples, made analytical errors” and mismeasurements. Gould concluded that these errors were a result of Morton’s racist bias.

DeGusta and Lewis go on to update the story with details of the recent effort by Lewis and collaborators to remeasure the crania studied by Morton and their finding that if anything, Morton overmeasured Egyptian crania (not white ones).

And therefore they write, in a passage that arguably summarises their thesis:

Gould was certainly right that all scientists, as humans, have some sort of bias. But while biased scientists are inevitable, biased results are not, as illustrated by Morton (biased) and his data (unbiased, as far as we can tell). Science does not depend on unbiased investigators but on methods which limit the ability of the investigator’s bias to influence the results.

Which raises the question of what method it is that permits the authors to generalise from a single [empirical] finding (the case of Morton’s lack of measurement error despite  the harm to his racist beliefs) to so general a claim as “biased scientists are inevitable, biased results are not” and their answer “Not so” to their own question “Is bias inevitable in science?” (it is worth noting that the title does not restrict itself to scientific measurements).

After all Gould does not claim that all results are inevitably biased. The authors do not provide any direct quotes from Gould in lieu of which I repeat the authors’ summary of Gould’s view: “Stephen Jay Gould claimed unconscious bias could affect even seemingly objective scientific measurements” [emphasis mine]; even they do not suggest that “Stephen Jay Gould claimed unconscious bias affects all objective scientific measurements“.

This is not, I believe, a trivial matter. The authors claim in a self-congratulatory tone that they (one of them) did the obvious thing that Gould had failed to do – to wit, remeasure the skulls. At the same time, they also claim that Gould’s thesis pertains primarily to “objective scientific measurements” or “actual measurements” [emphasis mine]. But if Gould did not actually remeasure the skulls, on what did he base his argument that the measurements were wrong? For this we can look to Gould’s original article:

Morton published all his raw data, and it is shown here that his summary tables are based on a patchwork of apparently unconscious finagling. When his data are properly reinterpreted, all races have approximately equal capacities.

It is worth quoting multiple sections from Gould’s original article to make the point to follow. Gould writes:

Morton … did supply one rare and precious gift to later analysts: he published all his primary data… I have reanalyzed Morton’s data and I find that they are a patchwork of assumption and finagling[.]

And:

[...] Morton’s method is suspect from the start for two reasons. First, he did not distinguish male from female skulls…. Second, he measured capacity by filling the skull with white mustard seed, sieved to reduce variation in grain size.

[Note: I am intentionally leaving out the substantive bits of Gould's argument in defence of his claim, since they do not pertain to the analysis here of DeGusta and Lewis's critique.]

What becomes clear in reading Gould’s paper is that contrary to DeGusta and Lewis’s summary, Gould was concerned with the measurement methodology and statistical presentation used by Morton. Having placed a thesis in their opponent’s mind that bias impacts “actual measurements“, DeGusta and Lewis are surprised that Gould did not then regenerate the “actual measurements“. But Gould was content to work with the “primary data” published by Morton – Morton’s “precious gift“. Contrary to DeGusta and Lewis’s apparent understanding, it should be clear from the quoted sections above that Gould’s claim is not that bias miraculously jumps from the scientist’s mind into the measuring devices and thence to the raw data.

Time and again, Gould in his paper mentions the problem of “finagling” (dictionary: “act in a devious or dishonest matter”) and it’s centrality. An issue dismissed to the margins by DeGusta and Lewis: “extreme bias cannot, short of fraud, influence the results” [emphasis mine]. By mentioning and dismissing the conscious act of fraud, DeGusta and Lewis ineffectively dismiss the “large middle ground” (to borrow a term from Gould) between conscious fraud and objective data.

DeGusta and Lewis end on a celebratory note on the greatness of science (due one assumes to its methodological strength, as they see it, which voids the need for examining such tricky issues as individual motivation). Gould’s derives a very opposite caution from his examples. In examining the Morton episode and the other examples (Newton, Mendel), Gould finds lessons for the scientific community:

I do share the scientist’s faith that “correct” answers exist for most problems, and I believe that fudged data are paramount as impediments to solutions. I only raise what I regard as a pressing issue with two hopes for alleviation — first … we may examine our own activity more closely; second, that we may cultivate, as Morton did, the habit of presenting candidly all our information and procedure, so that others can assess what we, in our blindness, cannot.

In opposition to DeGusta and Lewis’s optimism in the ability of scientific method to limit the effect of the bias of scientists, Gould’s significantly more nuanced position (read the paper linked to above) evinces (as philosophers of science have done before) both the prevalence of (and I would argue the need for) “finagling” or fudging in scientific work and the consequent need to be cognisant of the implications of this fact.

DeGusta and Lewis claim to debunk Gould’s claims of error in Morton’s data. If the remeasurement and reanalysis carried out by Lewis is correct, all that demonstrates is that Morton does not serve as evidence of Gould’s thesis, not that Gould’s thesis is incorrect. Particularly since Gould himself offers additional example of fudging of data by scientists, and many more are available: see the controversy surrounding Eddington’s measurements to purportedly confirm Einstein’s theory of general relativity.

Scientists take short cuts (for practical as well as necessary reasons). They base these shortcuts on their beliefs and commitments (ontological, epistemological, political, so on). Science advances because results are held to be provisional, not because “truth is … obtainable” or “science … is self-correcting“. DeGusta and Lewis offer nothing towards contesting this finding of historians and philosophers of science (including Gould), nor do they offer any explanation of how scientific method auto-corrects errors in such shortcuts (unless of course all they mean by scientific method is the mundane processes, not exclusive to science, of examining prior assumptions and conclusions, checking for errors, rigour, so on… none of which equals “self-correcting“).

 


Comments Feed
Comments
3 Responses

  • APR says:

    Very nicely done, in an attempt to hoist one (Gould) by his own petard, they show they neither know which petard needs to be doing the hoisting nor are they aware how much they are dangling from their own.
    In any event, they are effectively making an pre-Kuhnian argument in a world where Kuhn – for all the limitations of his work – is generally treated as the bedrock, the foundation, of modern science studies (from Lakatos to Latour, Feyerabend to Haraway). Any sense of where their insecurity comes from?

    • ravi says:

      Thank you for the comment Alan. You are right of course regarding the anachronistic nature of their argument. I suspect that is because the bedrock of science studies can be ignored because science studies can be ignored, in turn because science studies has been dismissed and shamed (re: Sokal). Muscular scientism! You see a very mild version of that in the recent polite spat between V. S. Ramachandran and Colin McGinn – http://l.ravi.be/nIUacx (or for that matter in the confident mutterings of “rogue” economist Levitt).

      I think I have a good idea where the insecurity comes from. Scientists and scientistic types are, I suspect, afraid of appearing soft-headed/hearted and not willing to “go where the evidence leads”. Hence Larry Summers on women, the whole EP programme, etc. Of course the strange thing is that they want to go where hard-headedness leads, not due to but despite the evidence.

      But wherefrom the confidence that philosophy and sociology can be utterly ignored? Perhaps the usual faith in reductionism? VSR truly thinks that mirror neurons can entirely explain civilisation! It helps that scientistic optimism of this sort is underwritten by a particular brand of philosophy epitomised in Dan Dennett, the Churchlands, et al.

  • ishi says:

    i had seen the PLOS article. one can remember the iq controvertsy—mismeasure of man. ‘does g exist’. was sjg’s analyses of principle component analyses/factor analyses hold up when he criticzed heritability studies? (see also jay joseph on wikipedia discussion of schizophrenia inheritance/twin studies). my view is sjg made some very good points but so do his critics and the controversy remains though (as in linguistics chomsky and his critics basically just ignore each other, have their own conferences and journals). sjg never responded to his critics really.

    there are better approaches to iq around but the old school usde old behavior genetics ideas which may be right but may not be.

    there is a ton of bias in science—most obvious when its corporate funded.

    but i dont get the point of this article. i’d have to look again at plos to see what they did and what they say sjg did.

    but my view its either right or wrong. if they lumped m and f skulls togther, thats obviously bogus. but i’d have to see. thats like measuring apples and oranges together.
    i couldnt really tell if you read the plos article’s ‘hard’ analyses. cheers.