For the most part, statistics is a method of investigation that is used when other methods are of no avail; it is often a last resort and forlorn hope. (M. J. Moroney, Facts from Figures)

Statistical analysis may be described as the quantitative treatment of uncertainty. In the broad sense, it is certainly more than that. To many people the term "statistics" is synonymous with "data" and a large portion of those who do statistical analysis concern themselves with collecting and summarizing data. But when data so treated are used to formulate and test hypotheses, probability is immediately involved and the quantitative treatment of uncertainty begins.

The malaise engendered when one deals with uncertainty and an insufficient knowledge of statistics probably account for the viewpoint expressed by Moroney. Many people, scientists among them, are uncomfortable dealing with uncertainty (even though, without being aware of the fact, they are constantly doing so) and their opinion of statistics is consequently somewhat colored.

We are interested here in whether or not statistical analysis of UFO sighting reports is likely to be informative as to what the phenomena are but not as to how they are reported. We make a distinction, initially, between studying the phenomena of UFOs and studying how people report UFOs. It is likely that the two cannot be completely untangled and, further, that the former is impossible without some idea of the latter. However, attempts have been made and probably will in the future be made to use aggregated sighting report data to study the UFO phenomena because that data source is certainly the largest and most comprehensive of any we have available with which to attack the problem. Throughout this chapter we will be concerned, then, with the role of statistical methodology in studying the UFO phenomena.

Since statistics deal with uncertainty it might seem an attractive candidate for a central methodology in UFO research. The purpose of this chapter is to discuss the place of statistical analysis in the study of the UFO problem. We will be specifically interested in the testing of hypotheses and with decision procedures and not simply in the aggregation of data.

The nature of the UFO problem coupled with the nature of statistical methodology, first of all, results in questions posed in the hypotheses which may not be particularly satisfying. For example, we might want to ask "Is there a 95% (or 90% or 99%) chance that UFO sighting reports include observations of objects not of terrestrial origin?" But by the nature of the data we are forced to ask questions such as "Is there a 95% (etc.) chance that the characteristics of reports classified as "knowns" differ from those for which no explanation has been suggested?"

One reason for the inability to ask questions or state hypotheses which are directed specifically at solving the problem of UFO phenomena is that they occur in nature and out of our direct control. Except perhaps for some psychological studies, we cannot place "the UFO problem" in a laboratory and measure and study it -- we must accept it as it happens. In statistical terms, we cannot design statistical experiments to test particular question.

The second, and more profound, difficulty is presented by the rather obvious fact that it is impossible to formulate meaningful statements, questions, or hypotheses about the manifestations of unknown phenomena. We can, of course, examine the data and see what manifestations there are in the sample data, but we are severely limited in the nature of the conclusions we can draw, again, because of the unknown nature of the phenomena. The difference here is subtle, perhaps, but important.

An instructive, but certainly not unique, way of looking at this difference is to invoke the traditional dichotomy between inductive and deductive reasoning in science. The deductive approach would operate by, say, assuming that UFOs are a manifestation of Extra Terrestrial Intelligence; or,. perhaps, simply represent a class of unknown atmospheric optical or electromagnetic phenomena. Given one or the other assumption it would next follow that some hypotheses about the characteristics of UFO reports be constructed. But because in both assumptions we are dealing with something unknown, how would we go about setting up such hypotheses? Such an approach from a statistical point of view at any rate seems so difficult to pursue as to be essentially valueless.

An inductive approach would, in this case, be something as follows. Let us aggregate a sample of UFO reports and examine their characteristics with the objective of establishing beyond some reasonable doubt that the characteristics are thus and so. From there we must try to build a theory which explains those characteristics.

Nearly all science operates in practice by a combination and alternation of inductive and deductive methods and in both statistics as a research tool is generally used. However there are some important differences in statistical method depending upon whether we look at that data or evidence in order to formulate a hypothesis or whether we wish to establish a degree of reliability for the validity of what we hypothesize. Perhaps the commonest misuse of statistics is represented by efforts to do both of these at once.

In statistical language, the expression of hypothesis formation after the fact, after examining the data, is called a posteriori hypothesis formation. The erection of a hypothesis before the data are examined is called a priori formation. The former follows rather easily as a result of the inductive approach and the latter from the deductive method. A posteriori hypothesis formation unless properly tested represents the previously mentioned attempt simultaneously to formulate a hypothesis and establish its significance.

In addition to the difficulties in hypothesis formation presented by the UFO problem, there is another problem which should be discussed. This problem, nearly always a crucial one and not as unique to the UFO problem as the one just mentioned, is the sampling problem. Granted that some hypothesis be formulated either a priori or a posteriori, we then must test the hypothesis on a randomly selected sample of data. We cannot enter into a complete discussion of random sample selection here, but must simply point out that if we hope to establish the true statistical significance of a hypothesis the selection of sighting reports cannot be biased either in favor of or against that hypothesis to be tested.

For example, let us suppose that we want to test the hypothesis that UFO sighting reports contain a significant (in some statistical sense) number in which the estimated apparent speed exceeds sonic or aircraft speed. Such an experiment could be set up and a sample of report data gathered on which to test the hypothesis. However, unless great care is used in selecting cases for inclusion in the sample, a non-random component is likely to be encountered. This is because it is very likely that it is precisely because the UFO exhibited what to someone was supersonic speed that it is reported and included in UFO files of one sort or another. Such a bias in the sample negates the possibility of a statistically reliable answer to the question embodied in the hypothesis.

The preceding example brings up a very perplexing problem. Just what should constitute the population of UFO reports? Should we include all UFO reports regardless of probable explanation, or just those reports for which no rational explanation can be given? It seems intuitively obvious that an observation which is almost certainly of, for example, Venus should not be included in the population of UFOs. But the possible dangers of biasing the sample of reports examined by such intuitive reasoning seem to be serious, to say nothing of the problem of determining the division between known and unknown cases. Again, it seems that the unknown nature of the phenomena poses some serious questions as to the definition of the population and therefore to the kinds of question we might ask of report data.

Some UFO literature has used aggregates of report data to search for "trends" or "patterns," either implicitly or explicitly stated.

The basic assumption seems to have been that trends and patterns in UFO reports might provide information on the nature of the phenomenon. This approach appears to be mostly inductive -- perhaps not surprisingly so in view of the difficulties in the deductive approach in the UFO problem.

There are two important comments on this assumption. The first is that any examination of report data is bound to turn up some pattern -- we would be quite surprised it the reports were completely featureless. The second is that, as already mentioned, since the patterns were detected from the sample in hand some procedure for testing the significance of the patterns on independent data samples is necessary.

The Vallees (1966) recommend a search for spatial and temporal patterns in the report data. They report:

1) a claimed tendency for report positions in a given calendar day to be located in patterns that can be joined by nets of straight lines (the controversial "orthoteny" hypothesis),

2) a difference in the diurnal variation of different types of UFO reports, and

3) a 26-month periodicity (adjusted for annual variation) in report data.

Only in the first instance do the Vallees report any test as to the statistical significance of the claimed pattern. They establish some basic criteria giving the distribution of the number of points determining straight lines used to join nets of points when the points are randomly distributed in space. They do not report, however, testing the straight line hypothesis on a data sample other than the one used to formulate the orthoteny hypothesis.

For the moment, let us assume that all three features may be tested according to the methodology of statistical hypothesis testing and any one proves significant -- that is, the null hypothesis of

1) a spatially random distribution of daily report locations, or

2) no difference in the diurnal variation of types of sightings, or

3) a temporally random distribution of monthly total number of reports

is rejected at, say, the 95% level. Therefore, we conclude with a risk of 95% that some non-random spatial or temporal variation occurs in sighting report data. This "risk level" is a measure of how confident we can be of rejecting the null hypothesis when it is in reality true. Most statistical tests are of this basic type.

However there is another type of statistical error which is inherent in this type of hypothesis testing which generally speaking should be taken into account. We should (if possible) try to determine what is the risk of accepting the null hypothesis when it is in fact false. Normally this type of error is guarded against by formulating the problem so that the status quo is represented by the null hypothesis. The rationale for this choice is that it is better to err on the conservative side, since generally the risk of accepting the status quo (null hypothesis) when it is in fact false is higher than the risk of rejecting it when actually true. The complete formulation of the problem in these terms would be an exercise in decision theory. Because of the interest aroused by the UFO problem, both scientific and social interest, it appears that a most interesting and appealing exercise would be an attempt to formulate some problems in terms of decision theory.

Even assuming that the decision problem can be attacked and solved and we accept the rejection of one of the null hypotheses, what have we learned? Obviously we are faced with strong evidence that there is something very peculiar about the distribution in space or time of sighting reports. But the use we could make of this peculiarity in drawing conclusions about the nature of UFOs would be limited because of numerous alternative explanations of a peculiar distribution of reports. Statistical reasoning in this hypothetical situation could tell us that the reports are significantly non-random in their spatial or temporal distribution and that the probability is large that there is something there to investigate, but statistical reasoning could tell us nothing about how to interpret this non-randomness. In addition the word "significance" is used in the statistical sense and has no connotation at all of "importance."

A useful analogy here might be the cigarette smoking-lung cancer relationship which has also been a storm center of controversy. The statistical significance of a relationship between the two has been established to be very high and almost everyone accepts the level of statistical significance as indicative of a relationship. However, this significance in no way proves a causal relationship between smoking and lung cancer -- that is merely one of a number of alternative explanations of the statistical result. Most people, in addition, would accept the significance level as evidence that there is certainly something to investigate. The use of statistical evidence to choose what to do next rather than to choose between terminal acts involves decision theory, rather than classical statistical hypothesis testing. This type of analysis has already been mentioned above.

To summarize, the UFO phenomena presents some difficult and challenging problems to statistical methodology. We are dealing with unknown phenomena, at least in part, which is manifested by subjective, qualitative reports from observers with a wide spectrum of ability to report what they see. We cannot place the phenomena in the laboratory to study them and design experiments on them. There are very fundamental problems such as defining the population to be used in statistical studies, and formulating hypotheses about characteristics or report data a posteriori and attempting to interpret these as manifestations of unknown phenomena.

The physical scientist conversant with statistics and statistical methodology is likely to come to one of two conclusions about the possibility of productive use of statistics in the UFO problem. Considering the difficulties described above he may conclude that the methodology of statistical analysis does not offer satisfying answers to the important, central questions of the UFO phenomena, and that efforts should be directed at increasing understanding of atmospheric optics, etc. or in attempting to make some measurement of some physical quantity associated with an UFO. Or he might take the position that difficulties of statistical analysis in this instance should not prevent efforts to make analyses, because the risk of throwing away valuable information by ignoring sighting report data should not be overlooked. This position must be taken with some care, however, for he would be taking it as "a last resort and forlorn hope" as Moroncy puts it.

The social scientist, on the other hand, might take a different position. Instead of concerning himself with report sightings as a measure of a physical phenomena he might be attracted by the data as a source of information on psychological and social-psychological problems of perception, reporting, etc. We do not regard ourselves as qualified to pursue this point further. Mention of it was made at the beginning of this chapter and additional discussion may be found in Section VI in Chapters 1 and 2.

As a result of considering the problem of the role of statistical analysis of report data in investigating UFO phenomena we conclude that very grave difficulties are present involving rather fundamental aspects of statistical methodology. It is our feeling that little value to the physical sciences will result from "searching" the report data for "significant" features.

We qualify this view in two ways: First, we are not able, of course, to perceive the future and it may be that an innovative worker paying careful attention to the demands of methodology might well produce a study which represents a real increase in knowledge about UFOs. We should in this regard give the decision-theory approach some thought: we should attempt to evaluate the consequences of statistical error of both kinds and to consider the problems posed by question of the "where do we go from here?" type. Second, efforts to investigate UFO reports rather than the UFO phenomena seem to offer fertile ground for future study.(1)

_________________

⁽¹⁾ The complete Condon Report can be found here.