E-Notes

Standards of Intelligence Reasoning

by Stephen Gale

November 14, 2003

Stephen Gale, Ph.D., is co-chairman of FPRI’s Center on Terrorism, Counter-Terrorism, and Homeland Security.

Reading the news these days, one gets the feeling that there is an air of true cluelessness as concerns the methods of reasoning used in intelligence circles. Take for example “The Stovepipe,” Seymour Hersh’s recent article (New Yorker, October 27, 2003) about the intelligence analysis leading up to the Bush administration’s decision to engage in the war in Iraq. Hersh (who is nobody’s fool when it comes to matters relating to the workings of government) repeats, in one form or another, the refrain that key information was supplied to— and analyzed by — senior administration officials (Bush, Cheney, Rumsfeld, et al.) “with little prior evaluation by intelligence professionals.” Hersh’s intent, I expect, is that we— those of us who are sufficiently interested in world affairs to have taken the time to read the article— will immediately recognize all that’s wrong with the Bush administration’s handling of the Iraq issue and foreign affairs in general: that it is far too politically motivated (read, directed towards the re- election of the President) and that momentous decisions related to war and peace should be based on information vetted by the highest intelligence standards (read, vetted by members of the professional intelligence community). Some, I expect, will be shocked by the so-called revelations in the article; other might be awed. But at least as I see it, few will have gotten the real story.

Just what is this intelligence business anyway? What is it that these “intelligence professionals” (the “intelligence communities” of the world) do to earn their keep? What are their professional standards, their backgrounds, training, risks and rewards? And, short of having worked in the intelligence field, how can we, as engaged citizens, decide whether to be shocked or awed by accounts that point to failures in political decisions that are based on intelligence analysis?

Intelligence gathering and intelligence analysis are undoubtedly complicated and weighty businesses. From Sun Tzu onwards, the safety and security of cultures, civilizations, and nation states have rested, in large measure, on the effective and efficient use of intelligence and intelligence analysis. As Sun Tzu put it in The Art of War:

“Raising a host of a hundred thousand men and engaging them in war entails heavy loss on the people and a drain on the resources. The daily expenditure will amount to a thousand ounces of silver. There will be commotion at home and abroad, and men will drop out exhausted.”

“Opposing forces may face each other for years, striving for the victory which may be decided in a single day. This being so, to remain in ignorance of the enemy’s condition simply because one grudges the outlay of a hundred ounces of silver is the height of stupidity.

“One who acts thus is no leader of men, no present help to his cause, no master of victory. Thus, what enables the wise commander to strike and conquer, and achieve things beyond the reach of ordinary men, is foreknowledge. Now this foreknowledge cannot be elicited from spirits; it cannot be obtained inductively from experience, nor by any deductive calculation. Knowledge of the enemy’s dispositions can only be obtained from other men.”

Historically, to know even rudimentary matters such as the direction to look for the coming sunrise required clear, unambiguous information, carefully analyzed, communicated in (as they say) a timely manner. Just imagine one of those great armies of the Arabian Peninsula in the 7th century looking to the Prophet for guidance on troop deployments at around two o’clock in the morning. It is pitch dark, no one quite remembers the terrain, and GPS devices are well over a thousand years in the future. Sound intelligence, primitive though this example may appear to be, was very likely a key factor in determining victory or defeat in the morning’s battle. Napoleon was supposed to have said “an army marches on its stomach.” To this he might have added Sun Tzu’s observation: “Hence it is only the enlightened and wise general who will use the highest intelligence of the army for purposes of spying and thereby they achieve great results. Spies are the most important asset, because on them depends an army’s ability to march.”

This is hardly the place to reconstruct the history of the value of intelligence in world affairs. What is of concern, however, is the issue raised by Mr. Hersh: Just what is gained by “the evaluation by intelligence professionals”? Where is the added value, the bang-for-the-buck that is supposed to be provided by Mr. Hersh’s professional intelligence analyses and evaluations?

As with so many of the journalists who specialize in the affairs of state, Mr. Hersh seems to be both in awe of the powers of intelligence analysts and shocked by the seeming reluctance of politicians to put those talents to good use. And, given the nagging problems that many now see with the conduct of US efforts in Iraq, Mr. Hersh could easily be justified in being shocked, awed — and possibly even angry. Hersh and others seem to feel that while Iraq may have been an easy military target, the aftermath - the on-going efforts to “rebuild” Iraq — is a disaster. What was the purpose of the war in Iraq anyway? If it was to eliminate Saddam himself, we did not get the right intelligence on his whereabouts and we still can’t find him. If it was to find and destroy all of those purported weapons of mass destruction — you know, the ones that the administration insisted were there but, apparently, never turned up — then Bush and his senior advisors obviously must have based their decisions on bad intelligence. And, at least according to Mr. Hersh, it was bad precisely because it had not been evaluated by intelligence professionals.

There are any number of problems with this perspective and, in the absence of some clarity on these matters, I fear that Mr. Hersh and others in his profession will continue to haunt us with tales of shock and awe rather than enlightening through the journalists version of “evaluation by intelligence professionals.” Admittedly, some of the problems are subtle; others are glaring; and still others are reminiscent of the fault that Mr. Hersh lays of the door of the Bush administration.

To get a fix on Mr. Hersh’s assessment of the Bush administration’s use of the intelligence process, we need to get some insight on just what intelligence analysts do— and are supposed to do. Although there are manuals that are used to train intelligence workers (most of which describe systematic, step-by-step approaches to the analysis and evaluation processes), as I see it there are really only four general characteristic standards and methods that are used in the practice of intelligence reasoning.

1. Ad Hominem Reasoning.

[1] In a manner similar to that of intelligence professionals, in the “The Stovepipe” Mr. Hersh employed a method parallel to one used in intelligence analysis and evaluation: the use of information derived from HUMINT (human intelligence) sources. Not being privy to the documented evidence of just what the Bush administration said or did in its deliberations on Iraq — and certainly having no firsthand experience in the administration’s councils — Mr. Hersh simply did what any good investigative journalist or intelligence analyst would do: he found a number of “credible” second and third hand sources who were willing to provide him with “raw intelligence.” And being one of the day’s premier journalists, Mr. Hersh undoubtedly followed in the steps of the best of the intelligence community by carefully scrutinizing and vetting each of his sources. Were they likely to have had access to the information? Are their comments internally consistent and are they supported by similar information from other sources? Do any of the sources have relevant hidden agendas that might have distorted their information? (Fortunately for Mr. Hersh, he did not have to trouble himself with a couple of the really knotty issues of intelligence analysis — translation and the human and financial costs of securing access to sources.)

But given his arguments about the poor quality of the administration’s sources, I am sure that Mr. Hersh appreciates that HUMINT is hardly infallible. Even more significant, Mr. Hersh’s HUMINT is probably fallible for precisely the same reasons that he was shocked by the Bush administration’s apparent use of “stovepiping:” the use of information that made its way directly to the top without pausing for “evaluation by intelligence professionals.” As with intelligence analysts, Mr. Hersh simply had to make do with those informants who were willing to speak with him and to evaluate their information in terms of his professional judgment concerning honesty and internal and external consistency. What is perhaps unique to Mr. Hersh’s work — in contrast with the procedures use by intelligence agencies — is that he needed to rely solely on his judgment (and, perhaps, that of his editors) rather than the more tortuous path required where a variety of intelligence analysts work from the same sources to produce a common evaluation. Methodological differences notwithstanding, however, in the end Mr. Hersh found himself in precisely the position of the Bush administration: on the one hand, needing reliable, accurate information and, on the other, needing to depend on often unreliable, inaccurate humans to provide it. Ultimately, both he and the administration appear to have resolved the problem in much the same way: by invoking ad (or, rather, pro) hominem claims — “urging from the person.” In Mr. Hersh’s case, it was all he could do. For the administration, I suspect that the rationale was much the same: they put their faith in the personal judgments of the senior members of the administration rather than the procedures of intelligence interlocutors.

2. Begging the Question.

In the eyes of an omniscient being, we are probably all intellectual beggars. Kant, in fact, spent much of his philosophic effort undoing the foundations of Aristotelian and Scholastic metaphysics in order to demonstrate that there could never be absolute reference points or archetypical categories of knowledge and understanding. So it is with evaluations by intelligence professionals: faced with a mountain of information of variable worth, intelligence analysts must beg just those questions that they are responsible for answering.

Take the case of the report by the Italian intelligence agency SISME on the possibility that Niger had arranged to sell Saddam yellowcake (the precursor to the fissionable materials used in nuclear power plants and bombs). [2] Unquestionably, the report was dated, incomplete, and potentially at odds with the information available from other sources. Neither was the report based on first hand documents (e.g., invoices, bills of lading, delivery dates). Rather, it was simply a rough indicator that something untoward may have happened between Iraq and Niger related to the sale of yellowcake. As Mr. Hersh sees it, Bush and his senior advisors chose to beg the question: given the administration’s long-standing concerns with respect to Saddam’s ability to construct and deliver nuclear weapons, given their deep suspicions of Saddam’s motives, given their reluctance to believe that Saddam had willingly dismantled the research and production facilities that were known to exist prior to and after the Gulf War, and given their fear of being blind-sided by an Iraqi nuclear weapon in much the same way much as we missed signals related to al Qaeda’s operations prior to September 11th, the administration begged the question by assuming that Iraq was still attempting to build nuclear weapons and then using the SISME report for support.

I think that we would all agree that the evidence provided by the SISME report did not conclusively establish the Iraq-Niger connection. At the same time, I also seem to recall that, since September 11th, there has been a sizable cottage industry devoted to pinning the blame for the hijackings on the fact that the Bush administration had failed to thwart the attacks precisely because it had not begged the question — that it had not evaluated and connected the dots. Dots? What dots? The amateur “red team” efforts by a handful of security experts and novelists? The assortment of disconnected incident reports filed by a few FBI agents who, under normal circumstances, were expected to provide documented evidence supporting existing (or sometimes even future) criminal actions? Without begging the question— without assuming that al Qaeda was preparing to strike at US domestic targets, that it was prepared to organize and execute to actions that potentially could have successfully decapitated the US government with a one-two punch, and that al Qaeda had already deployed operators within the US — those supposed dots had little more credibility than the questionable information that Mr. Hersh cites from the SISME report.

All intelligence analyses “beg the question” since, by virtue of its mandated role, intelligence analysis always resorts to the construction of possible scenarios and moving around the puzzle pieces to determine whether there are any indications that one or another of the scenarios is potentially becoming fact. Intelligence reasoning does not proceed by the so-called scientific method where the game is to test the validity of theories by a proposing hypotheses and testing outcomes (usually by predictions) in controlled experiments. Nor does intelligence analysis employ legal reasoning, the standards and methods so dear to the hearts of our nation’s myriad lawyers: evidentiary rules, adversarial questioning, judgment by one’s peers, statutes and case law, and all the rest simply have no analogues where hard evidence is hard to come by, the “accused “is not available for questioning, and conclusions are arrived at without direct reference to standards that are easily appreciated by a man in the street. Rather, good intelligence reasoning relies on begging the right questions, recognizing kernels of meaning in obscure patterns, and (in most cases) making the right choices before - not after - the action has occurred.

3. Betting the Odds.

Much as we all hate to admit it, there are times when, even under the best of circumstances, we are wrong. Worse still, we are often wrong-headed. In most cases, of course, being wrong results only in mild embarrassment. And even when we are wrong-headed, we can often rely on experience, education, and the expectation of rewards to help us to change our minds.

Intelligence professionals — at least in their professional capacities — do not often have the ordinary luxury of suffering mild embarrassments (losses) without incurring the potential of extreme costs. Unlike the businessman who does not close a sale or two, failures in intelligence analyses and operations usually result in large-scale strategic — not simply tactical — losses. The failure to properly recognize the value of one “dot,” for example, could easily lead to the misinterpretation of the potential connections to and an entire series of actions. And the failure to quickly recognize the meaning of a pattern can easily result in delays that are unrecoverable.

The full list of failures of the power of reasoning in the intelligence community can probably never be known but, to keep matters in perspective, just recall some recent examples: Pearl Harbor, the first Soviet nuclear test, Sputnick, the Cuban missile crisis, the Tet offensive, the fall of the Shah, the attack on the Marine Corps barracks in Beirut, and (of course) September 11th. A mixed bag with mixed results and yet a mixed bag each with highly consequential outcomes. All were failures of intelligence reasoning (some with greater or lesser import), but each with dramatic — often long-term — consequences. In a sense, the failures of intelligence reasoning are far more similar — at least in scope if not in kind — to the failures of CEOs or cardiac surgeons than to the failures of the sales force or chiropractors.

That being said, what can intelligence professionals use as standards for evaluation? Ritual computations of net present value and tracking indicators such as stock prices and price-earnings ratios have historically carried little operational weight in intelligence analysis. (Recall that Defense Secretary Robert McNamara actually tried to use methods of this type — without much success!) Other standards — focusing on perceived key issues (hot tips), tracking historical trends (business cycles), and balanced global even-handedness (portfolio management) haven’t had much success either. When it comes right down to it, about the only standards left for the intelligence professional is to bet the odds — say, much as property-casualty underwriters (as opposed to actuaries) do in the evaluation of insurance risks.

Lloyds London, the first of the world’s premier property-casualty insurance underwriters, would probably be the first point out that betting the odds is tough work. Worse still, it is very risky. Based on a confluence of information, projections of the occurrence of natural events, and maybe even hunches, property-casualty underwriters work their magic through pattern recognition and ensuring that the worst-case scenarios are thoroughly reviewed and factored into their calculations. Recent information systems improvements have assisted with the work on the front end (information management systems) but, in the end, it is the underwriters’ judgments that set the odds. For the intelligence professional, the methods employed are much the same — but, of course, the tolerances are different. Failures for an underwriter are measured in financial losses; failures in intelligence reasoning are measured in historical shifts.

4. The Rule of Complexity.

All too often, supporters of the scientific method invoke homilies such as Occam’s Razor to support their directives favoring simpler over more complex solutions. Independent of the subject matter or the degree to which matters have been codified, those favoring simplicity in fields outside the natural sciences often do so on mere principle rather than in view of the contexts and circumstances. Now, there are many cases in which simple explanations have won the day. But particularly in those cases where understanding refers to social behavior, organizations, and institutions, there are just as many cases where its evil twin, being simplistic, has undermined the rule of simplicity.

Hard and fast rules are difficult to find in the world of intelligence analysis. More often than not, intelligence analysts are faced with highly complex circumstances, imperfect information, obscure motivations, conflicting interpretations, and deep ideological and political differences. Simple answers are not only in short supply but, when invoked, have the downside consequence of being misleading or worse.

To its credit, the Bush administration was reasonably well aware of the dangers of relying on simple insights and solutions: Bush and his team have been endlessly skewered with respect to their (supposed) intelligence failures prior to September 11th. The administration was also well aware that traditional intelligence methods had not been all that accurate in the past, that events on the Iraq scene were probably moving too quickly to be able to rely on the kinds of cookbook procedures fashionable to today’s intelligence circles, and that, whatever else might be the case, all matters relating to the conduct of affairs in Iraq were likely to be very complicated indeed. Ever since the end of the Gulf War in the early 1990s, the nation’s political leadership has been arguing about the depth and extent of the Iraqi threat and all have concluded that matters were far too murky to permit clear answers. In effect, the Bush administration’s vision was that matters were complex — and made more so by the Saddam’s use of disinformation as a political weapon. [3]

Fast forward now to the post-September 11th, post-Afghanistan view of the Middle East. Saddam is still in power, the earlier weapons inspections were never completed, al Qaeda appeared to be reasonably healthy, and there was little hard evidence to support one or another position on the prospects of Saddam’s interest in using weapons of mass destruction. If anything, matters looked reasonably ominous: al Qaeda seemed to be gaining support, Saddam was making his usual threats, and Iraq’s December, 2002 report on its current weapons programs appeared to highly suspect. From the Bush administration’s perspective, the question “Whom can we trust?” was clearly problematic. Certainly not Saddam’s word. Probably not the step-by-step intelligence analysis procedures that had missed so many critical events in the past (either through weak analysis or less-than-timely results). In this climate, the administration’s response was predictable: accept that the situation is complex and not likely to be cleared up in the short-run; accept the short-comings of the intelligence community; and use the best available group of informed analysts to make the initial assessment of any new information. That the analysts were not designated “intelligence professionals” only speaks to the workings of the intelligence bureaucracy, not the quality of the administration’s intelligence reasoning.

Surely, political considerations played a substantial role in the methods used by the Bush administration. After all, that is precisely what we bargained for in our Constitution - that the nation’s political leadership is responsible for the hard decisions. Did the Bush administration’s political biases mislead their thinking? Possibly — but, then again, probably no more (or less) so than any other administration. What the Bush administration did was only to attempt to minimize the complexity of the information flow and analysis process by bypassing steps that, to the administration’s way of thinking, were not working all that well to begin with. Had the administration been correct, had our troops found Saddam’s weapons factories and caches (notably based on a CIA report on the purported locations of Iraq’s weapons facilities), their intuitions regarding the SISME report might have been faulted on procedural grounds— but probably only after we, as a nation, hailed the foresight and perseverance of Mr. Bush and his advisors.

As I see it, the matter of the SISME report is still very much up in the air. The situation in Iraq was always complex and continues to be so — even now. The evidence presented by Mr. Hersh has the ring of truth but, then again, so do other interpretations of the same evidence. It may well be that someone in Niger did discuss the sale of yellowcake to Saddam, but that those involved did so without the government’s direct involvement. It may also be that the sale was consummated, but that the material was never shipped - a sort of get-rich-quick scheme by one or another group in Niger. As with so many other cases in the intelligence field, matters are simply too complex to give all that much weight to what appears to be an open and shut case. Complexity, after all, is the rule rather than the exception in intelligence reasoning.

The items on the list could easily treated in greater detail, but my guess is that I’ve reached a point of diminishing returns. As matters now stand, Mr. Hersh seems to have done all of us a good service by recounting some of the facts that he believes we should consider in grading the Bush administration’s homework on the intelligence analysis used to justify its decision to move forward with operations in Iraq. Should we take Mr. Hersh’s argument as the final word? Clearly not, since much of his discussion rests on many of the same problems he finds with the Bush administration. Is it an interesting argument? Sure— but, again, only if taken in the context that Mr. Hersh probably does not have much of a handle on the complexities of the situation.

What we all need to keep in mind is that intelligence gathering, intelligence analysis, and intelligence evaluation has few parallels to the type of cookbook processes that Mr. Hersh appears to want us to believe in. Just as intelligence professionals have gotten it right in the past, just as often they have also been colossally wrong. To invoke their guiding spirit as the answer to the administration’s apparent shortcomings only does a disservice both to the intelligence process and the value that our Constitution places on political guidance and leadership. Were intelligence reasoning easy, I suspect that we would have seen competitive databases and analysis software vying for the intelligence market. That this has not been the case speaks more to the fact that this is a world of ambiguity and complexity than one certainty and simplicity.

Ultimately, we must also be clear that, regardless of what Mr. Hersh or some intelligence professionals would like to have us believe, the Constitution vests the power of both political and military decisions in the elected President. Of course, the President may choose to look for insight and support from any of a variety of sources (including intelligence professionals) but, at least as matters stand today, authority for foreign policy rests with the elected office— and not with the relatively ambiguous standards of reasoning employed by intelligence professionals.

Notes

[1] Strictly speaking, this discussion is more about pro — rather than ad — hominem arguments. Ad or pro notwithstanding, the strength of all such arguments amount to arguing from the credibility of the witness, not the facts.

[2] “Follow the Yellowcake Road.” Newsweek, July 23, 2003.

[3] See Kenneth M. Pollock’s The Threatening Storm (Random House, 2002) for a comprehensive discussion of this issue.

You may forward this email as you like provided that you send it in its entirety, attribute it to the Foreign Policy Research Institute, and include our web address (www.fpri.org). If you post it on a mailing list, please contact FPRI with the name, location, purpose, and number of recipients of the mailing list.

If you receive this as a forward and would like to be placed directly on our mailing lists, send email to FPRI@fpri.org. Include your name, address, and affiliation. For further information, contact Alan Luxenberg at (215) 732-3774 x105.