*This is the third installment of Sam Ponders Epistemology.

Sam Seitz

How can one assess the quality of an expert in a field in which one is a novice? To phrase the question somewhat differently, when two putative experts disagree, how is a layperson to know which is correct? This is a pressing and practical question for those interested in epistemology, and it is of particular salience now given the ever more specialized world that we inhabit. Indeed, while a vast number of decisions today demand scientific understanding, we cannot seem to escape the distorting effects of pseudoscience and conspiracy theories. One interesting approach to resolving this problem has been developed by Alvin Goldman, who sets up five criteria by which he claims a complete novice can assess the credibility of experts and, by extension, the accuracy of their claims. Goldman’s exact criteria will be described below, but fundamentally his argument is “that your evidence about the properties of the speaker is crucial evidence for your overall entitlement to accept the speaker’s assertion.”[i] Unfortunately, while this claim is likely correct in a general sense, it fails to resolve the novice/expert problem specifically, as one cannot evaluate the properties of a speaker without already having some familiarity with the topic on which the speaker is presenting.

To clarify this criticism, it is helpful to take a closer look at the exact criteria established by Goldman and then consider some counterexamples. Goldman argues that, when evaluating competing expert claims, a novice should consider five criteria, which I have paraphrased:

  1. Which of the experts presents the more logical, well-argued, and evidence-based claims?
  2. Which of the experts presents claims supported by the broader community of experts?
  3. Which of the experts can claim support from meta-experts (gatekeeping institutions like degree-granting universities, academic journals, etc.)?
  4. Is there evidence that either of the experts have biases or financial interests regarding the issue at hand?
  5. What are the experts’ past track-records?[ii]

As previously noted, these questions can help one determine which putative expert’s claims to trust, yet several problems remain. For one, Goldman never offers a hierarchy by which to prioritize these criteria: What should a novice conclude if one expert makes a more cogent and compelling argument but the other expert’s argument resides on the side of the scientific consensus? However, there is a much more fundamental problem, which will be the focus of the rest of this paper. Namely, it is impossible for a novice to utilize these criteria without some baseline understanding of the subject matter. One cannot evaluate how logical an argument is, whether meta-experts are legitimate, or how accurate an expert’s past claims have been without some prior knowledge of the subject. To his credit, Goldman acknowledges this problem in his framework, but his responses to this objection are fairly uncompelling.

Regarding the first criterion, he distinguishes between direct argumentative justification, which relies on the content of arguments, and indirect argumentative justification, which relies on an expert’s “dialectical superiority.”[iii] By this, Goldman means the expert’s general form, confidence when speaking, and ability to answer rebuttals. But this does not seem to be a compelling solution at all, as one’s ability to speak well is not evidence of the verity of one’s claims. And while the ability to respond to rebuttals is slightly more revealing, a novice would likely lack the requisite knowledge base to know if the rebuttals were grounded in truth or simply nonsense fabricated on the spot by a skilled orator. It is also not clear that a novice would even know whether a rebuttal was addressed if the response was non-intuitive and replete with jargon. In other words, an abstruse or overwrought explanation might be misunderstood and thus deemed by the novice to be a non-answer.

Goldman also recognizes that the assent of other experts and “meta experts” is not always a perfect indicator of accuracy, as they may simply be the victims of a cult-like dynamic in which they are so impressed by a putative expert that they lend their credibility without cause or reason.[iv] In a slightly weaker version of this recognition, Goldman suggests that members of the scientific community might simply believe “rumors” or claims without independently verifying them.[v] Obviously, if this were the case, having the support of “meta experts” or a scientific consensus would mean very little. However, his response to these possibilities is unsatisfying, as he simply defines away the problem by asserting that genuine experts would not behave in this manner.[vi] My personal experience in political science suggests this assertion is simply not true. I have encountered many experts with PhDs who often make inaccurate assertions about subjects that may lie on the periphery of their field of study but whose credentials in the broader field lend a false imprimatur of credibility. But even if my experience is unrepresentative, there still exists the deeper problem that determining the credibility of “meta experts” and the reliability of the methodologies used by the “consensus of experts” requires specialized knowledge. An expert could have a Ph.D. from a non-rigorous degree mill and have been published in low-tier journals without peer review, granting him the trappings of “meta expert” endorsement. But without some familiarity with the field, a layperson wouldn’t know which institutions and journals are credible and which are not.

The conflict of interest question is less vulnerable to this problem. However, it is worth highlighting that it is at least equally likely that an expert invests in a product she thinks is based on good science, and thus likely to be profitable, as it is that she distorts science for financial gain. Moreover, one can be biased and still conduct first-rate research. The final question, which concerns an expert’s track record, is perhaps the most vulnerable to my critique. This is troubling, as Goldman considers this criterion “the novice’s best source of evidence for making credibility choices.”[vii] But someone with no knowledge of a field has no basis for evaluating an expert’s track record, particularly when the predictions are nuanced and complicated. A novice may also not know how subfields interact and, therefore, be duped by an expert’s track record. For example, a commentator discussing immigration may have a great track record in finance but know nothing about labor economics. Thus, his track record in “economics” is irrelevant, but a novice unaware of the different fields within economics may not appreciate this distinction.

So, how might we rescue Goldman’s position? After all, we all need to rely on experts, and the criteria Goldman proposes seem intuitively appealing, at least to some degree. Goldman himself offers one suggestion, which is to use exoteric knowledge accessible to laypeople in order to evaluate esoteric claims of experts.[viii] The example he gives is an astronomer’s prediction of an eclipse based on complex celestial movements.[ix] Someone with no understanding of the solar system cannot look at the equations and models and evaluate them, but she can witness an eclipse and, through this exoteric knowledge, formulate a judgment about the astronomer’s prediction. The problem is that we do not always have the luxury of waiting for esoteric claims to become exoteric. But the bigger problem is that not all predictions are so easy to evaluate. An economist might predict a future unemployment rate and then, when the time comes, announce that she has measured unemployment to be as she predicted. But for a layperson who does not comprehend her methodology and lacks the ability to verify her unemployment measurements, there is no exoteric way to evaluate her claims; he must simply take her at her word.

Another solution would be to expand Goldman’s exoteric/esoteric distinction and argue that laypeople can extrapolate from areas in which they are knowledgeable to assess expert claims. An individual could, for example, observe her friends’ and colleagues’ employment status to guess unemployment levels. The problem is that this guess would likely be rough at best and completely wrong at worst, and thus not an effective way to vet an expert claim. But it gets worse because most forms of specialized knowledge don’t become exoteric. If an astrophysicist announces that a star’s dimming proves the existence of an exoplanet three times the size of earth, no layperson has or will have any knowledge by which to evaluate the claim. Of course, one can define “novice” more generously than Goldman and allow some kind of specialized knowledge – for example, a non-physicist who nonetheless understands high-level mathematics and uses this knowledge to assess the relevant equations – but it seems as though at this point it is more of a case of an expert evaluating the work of another expert.

Ultimately, it appears that Goldman must amend his standards and say that a novice must utilize his criteria and have some basic knowledge about the relevant field so that he or she at least has a general understanding of the arguments. In other words, one must have some expertise about a topic one hopes to evaluate, but expertise should be defined along a spectrum. One need not be the foremost thinker in a field to evaluate expert claims, but one must have some degree of field-specific knowledge. Nobody will ever be able to verify claims from experts in every field. However, by embracing intellectual curiosity and seeking to acquire a basic understanding of concepts that seem important or salient, we can generally acquire enough baseline knowledge to intelligently consume and evaluate expert claims. This kind of general knowledge would allow Goldman’s standards to be applied more critically and thoroughly, thus making them a more potent tool for one to discern among the competing claims of experts.

________________________

Notes:

[i] Alvin I. Goldman, “Experts: Which Ones Should You Trust?” Philosophy and Phenomenological Research 63, no. 1 (Jul. 2001), 88.

[ii] Ibid., 93.

[iii] Ibid., 95.

[iv] Ibid., 98.

[v] Ibid., 99.

[vi] Ibid.

[vii] Ibid., 106.

[viii] Ibid., 94.

[ix] Ibid., 106.

Bibliography

Goldman, Alvin I. “Experts: Which Ones Should You Trust?” Philosophy and Phenomenological Research 63, no. 1 (Jul. 2001): 85-110.