96. "Strong Inference and Weak Data"

SUMMARY

There exist alternatives to the use of mathematical statistics in drawing inferences from data. One of these is called strong inference. Strong inference gets its name because all it takes is a viable counterexample to a theory to create doubt about its truth. A viable counterexample means that the current statement of a theory is in need of improvement.

Strong inference is basically a simple idea: Conduct a study that is likely to produce an empirical counterexample to a theory, accept the conclusion that the theory (any theory, including and, especially, one's own), as written, will always need improving, and make the necessary theoretical improvements to eliminate the counterexample. Strong inference is based on the robust conclusion that eventually all theories are proven wrong, in need of restatement, or are ignored. That being the case, working to improve a theory is preferred to justifying it. The best way to improve a theory is to find out where it breaks down. The flaw, once identified, can lead to a new theoretical statement to remove the flaw. But the revised theory will also be incorrect and so the process of strong inference proceeds in an endless succession (hopefully) of increasingly less incorrect theories. Strong inference is a multistage, on-going process for discovery.

The article notes some disadvantages to the use of the strong inference process. Having noted them, the article concentrates on why strong inference is necessary and how one can engage in the use of strong inference processes. One enduring problem in our field is the weakness of our data. There is the risk of accepting a number describing some property of a system as fact. The main fact about facts is that they should not be trusted unless one knows precisely how they are generated.

Unfortunately, many organizational researchers rely on Other People's Data and Other People's Instruments for their data. Such data are contaminated by unknown decisions and assumptions over which the researcher has relinquished control. These data force one to deal with inherently contextual data without knowing the context.

Strong data have well-defined conceptual, measurement, and experimental contexts. Weak data fail to meet the standards of strong data. Stronger data is preferred to weaker data but often one is limited to the use of weak data. Strong inference is ideal for the discipline of successively strengthening weak data.

The article contains a discussion of how "results" of a study are outcomes of a lengthy sequence of transformations. For example, one has to select the arena (field study, lab study, archival study, etc.) for conducting a study. Then one must chose which data to observe and record. Then the observed data must be coded. The coded data are then encapsulated into measures. The measures are incorporated into models to produce an advanced form of data, that which is ready for hypotheses testing. Finally, the data is subjected to an inference procedure to produce "facts" or "results."
An experiment is defined in this context as including a pair of pairs with each pair being defined with respect to a consistent set of transformations. The pair of a theory and its predicted results is matched by another alternative theory and its predicted results. The two sets of predicted results ideally are disjoint. Hence, if a study is conducted and no counterexamples are found, then the theory is "not yet incorrect" but the alternative theory is rejected, as written. And if the study produces counterexamples, the theory, as stated, is rejected and the alternative theory is not yet incorrect.

The strong inference process involves multistages. The conclusion at each stage can be represented by a node in a strong inference tree. This article provides an example of a strong inference tree and details a procedure for evolving such strong inference trees. The described procedures for strong inference have been employed in a variety of settings and for a wide range of organizational and psychological processes.

It is important to understand that it is, in most instances, just as problematic to know what is false as to know what is true. However, viable counterexamples are relatively clear. A counterexample is viable if (a) after allowing for measurement error, a datum still lies within the set of results that disconfirm the stated theory, and (b) it is possible to replicate the conditions producing the counterexample. Counterexample, however, should not be considered definitive because the strong inference process continues. The strong inference process depends on a pair of propositions: (1) the theory is always incorrect, and (2) any inconsistencies in the results of research are due to a failure of the theory. The first proposition is quite reasonable. The second has two purposes: to focus attention on the theory rather than on the transformations and to be critical of petty excuses and eclecticism. The proper place for the burden of proof should be on the theory and not upon the cleverness of ex post facto data manipulations.

The article then explores the relationship between strong inference and processual research. It concludes that strong inference is ideally suited for conducting processual research. Strong inference is also used informally in management.

A version of strong inference is played out daily on an informal basis when there is management conflict. The challenger or rival uses the rhetoric of strong inference to create doubt on a practice or policy by focusing on inconsistencies. The defense uses a more forgiving rhetoric regarding contrary information. It might, for example, admit to the problem but state that overall it is immaterial. Defense is more statistical and challenge is more strongly inferential. The process of thinking along the line of strong inference is seen as the means for avoiding really stupid mistakes, to prevent disasters, and to overcome strategic blinders. Strong inference thinking helps avoid the Type III error of deriving a good solution to the wrong problem. CEOs are awash in weak data. One of their main jobs is to convert weak into strong data by a sequence of successive applications. The spirit of strong inference helps one gain the appearance of wisdom and promotes courage by encouraging acting in the face of doubt.