Philosophy of History Part XXV: Karl Popper and Prophecy in the Social Sciences

The belief in historical destiny is sheer superstition. There can be no prediction of the course of human history by scientific or any other rational methods. –Karl Popper

Karl Popper (1902 – 1994) was one of the most famous, and probably the most influential, philosopher of science of the twentieth century. He grew up in the Vienna of Freud, Wittgenstein, the Logical Positivists, and the young Adolf Hitler, and in many ways his philosophy can be understood as a response to its culture of high intellectual life and political fragility. His best-known work, The Logic of Scientific Discovery, was published in 1934, and aimed to clarify just what it was about science that made it special. He returned to this theme with The Open Society and Its Enemies (1943), and then The Poverty of Historicism (1957)—his contributions to the Second World War and the Cold War, respectively—in which he set out to show that the Hegelian and Marxist views of history were both tyrannical and logically incoherent. Since The Poverty of Historicism refines and systematizes arguments first presented in The Open Society and Its Enemies, it is on that text that we will concentrate.

Popper defined the object of his criticism as historicism: “an approach to the social sciences which assumes that historical prediction is their principal aim, and which assumes that this aim is attainable by discovering the ‘rhythm’ or the ‘patterns,’ the ‘laws’ or the ‘trends’ that underlie the evolution of history.” He divided it into pro- and anti-naturalist doctrines, according to whether their partisans affirm or deny that physics provides an appropriate model for the social sciences, and set out to show that both were logically incoherent and politically tyrannical.

According to Popper, pro-naturalist historicists (i.e., Auguste Comte, Karl Marx) argue that sociologists aim at theoretical history, derived from the empirical study of actual history, and at general, long-term predictions after the model of astronomy. Because the aim is to predict change in the future, the appropriate evidence is found in periods of change in the past—or, in other words, of revolution. Such predictions may be considered under two headings: “prophecies,” which are beyond the power of human agency to avert or bring about, and “technological predictions,” which describe the likely consequences of a given activity. Prophecy denies what technological prediction affirms: that human agency is the decisive element in history. Where technological prediction advocates social engineering in order to realize limited but definite aims, prophecy advocates preparation for the inevitable in order to turn it to advantage, or else minimize the resulting harm. Significantly, technological prediction is the characteristic method of physics, which, Popper argued, pro-naturalist historicists simply misunderstand. Historical prophecy implies “the historicist moral theory,” that “the morally good is what is ahead of its time in conforming to such standards of conduct as will be adopted in the period to come.”

Pro-naturalist historicism can be criticized in the following ways. It holds that history is governed by “laws of evolution” which determine long-term historical outcomes. However, there can be no “law of evolution” because natural selection, and like biological hypotheses, are not causal laws, but descriptors of general trends, composed entirely of contingent and unique historical events. If historicists object that living things have life cycles, and that these imply causal laws, and that civilizations also seem to have life cycles, they can be answered by pointing out dissimilarities in circumstance, and by explaining the perceived similarities as instances of confirmation bias. If they object that even a set of unique processes may reveal trends, and that these suggest causal laws, this may be admitted. What may not be admitted is that the trends themselves are laws. For instance, it would be erroneous to argue that there exists a historical law of increasing social complexity, just as there exists one of increasing biological complexity, because these are both instances of contingent trends, which could be arrested or reversed in a way that Boyle’s law, or Planck’s constant, which are causal laws, could not. The difference between trends and laws in sociology is the difference between technological prediction and prophecy: the former asserts that under a given condition, a given result may be expected, whereas the latter asserts that whatever conditions may be, a given result is certain.

Further, the true scientific method (falsification) is applicable to sociology, and it is the failure of historicists of both camps to adopt that method (the one by denying that it is applicable, the other by misapplying it) that accounts for the poverty of their discipline. The study of man, society, and the past can be considered under two headings: actual and theoretical. The actual disciplines, such as history, are concerned with the explanation of singular events through causal laws. These in turn are provided by the theoretical disciplines, such as sociology. In consequence, the historicist doctrine that the origins of a thing explains its later development is mistaken. It is the non-contingent laws that govern both the origins and the later development of that thing that sociology should concern itself with.

The goal of history and sociology to offer a scientific explanation of man, society, and the past, is both legitimate and urgent. Historicism is invalid not because it applies the methods of science to these subjects, but because it either fails to apply them, or it misapplies them. Only a positive program, superior to historicism, can displace it. Because the nature of historical explanation is such that there is always more information, and more causes at work, than can be comprehensively analyzed, historical science is of necessity selective, and therefor a matter of interpretation. Historians are not obligated to eliminate interpretation from their work, as both the historicists and their predecessors think, but rather to explain their point of view clearly, so that others may judge the quality of their interpretations for themselves.

Anti-naturalist historicists (i.e., Oswald Spengler, Karl Mannheim, John Stuart Mill) argue that each period is governed by particular historical laws, that these are not applicable between periods, and that in consequence any attempt at generalization for the whole of human history is futile. Social experiments cannot be conducted after the manner of physics in consequence of the scale and complexity of the subject, the novel and unique character of its processes, and the impossibility of separating the researcher from the process. “Exact and detailed” predictions in the social sciences are self-refuting, because that knowledge would alter the circumstances on which they were based. That knowledge could not be suppressed, since the method that produced it would be rediscoverable by anyone. Society possesses an organic unity that transcends the structure of its components, and makes it qualitatively different than the objects of physics. In consequence it must be studied holistically, and not through the disintegrative methods of physics. This in turn suggests the organic (totalitarian) theory of man and society. For holism intuition and analogy, not experiment and observation, are decisive.

Holism is derived from the philosophy of Plato, who regarded ideas as the fundamental reality, and whose influence on historicism is evident in the search for supersensory entities (or essences—e.g., race, class, society, History, etc.) that condition and explain sensory phenomena. Thus the study of history is, for holists, the study of those entities, and not of the sensory phenomena that they produce. This method is opposed by the empiricists, who regard sensory phenomena as the fundamental reality, and ideas as merely arbitrary and insubstantial descriptors thereof. Social science is thus properly conceived as the study of sensory phenomena (or objects—e.g., individuals, documents, artifacts, etc.) and not of super-sensory entities, which exist nowhere but in the minds of people who believe in them. Empiricism is the superior method, as the success of physics and biology, where it is taken for granted, demonstrates. The inability of social scientists to replicate the success of physics and biology in their own field is largely explicable in terms of their holist methodology.

Anti-naturalist historicism can be criticized in the following ways. Inattention to practical social problems has led to an overly theoretical approach, which is disconnected from, and of no practical use for the resolution of, those problems. While pure research certainly has its rights against the more pragmatic variety, the latter often serves to check and inspire the former. Properly speaking, they are partners rather than competitors. The object of legitimate sociology is “piecemeal social engineering,” i.e., planning informed by technological prediction, pursued with caution, and aimed at achieving limited and definite results. This is in opposition to the “utopian social engineering” of anti-naturalist historicists, who aim to “seize the key positions,” and use these to radically restructure an entire society in accordance with some vision or ideal. Because it is impossible to restructure an entire society at once, in practice utopians must employ the more limited methods of piecemeal engineering. However, their recklessness aggravates the dangers of this method and deprives it of its virtues. At the same time, unintended consequences, which are difficult enough to account for under a piecemeal system, become more numerous and more severe under the utopian plan. These in turn require additional planning to account for, which must produce additional unintended consequences, and so on. In short, the plan gets away from the planners.

A further problem with this method is that it seeks to control human behavior, and ultimately to produce a new kind of person, i.e., the kind who is content, pliable, and enthusiastic. The original aim of the utopian engineer (to create a society fit for people) is thereby subsumed into a new aim (to create people fit for society). However, this transformation destroys the scientific character of the project, since discontent, the one empirical test that the first aim admitted, can now be reinterpreted as evidence that further planning is not futile, but necessary. Holist methods are invalid because interrelations between components, and the perspectives from which they can be viewed, cannot be exhaustively described, nor can they all be taken in at once. Evidence must be selected before it can be interpreted, and this in turn suggests atomistic and empirical methods. Further, holistic plans of restructuring face the difficulty of infinite regress, since any activity intended to control the totality of social relationships will create new social relationships, which lie beyond the scope of the original plan, and which will require new plans and new activities to account for, and so on ad infinitum.

Anti-naturalist historicists argue that social experiments must encompass the whole of society in order to account for all variables. However, such experiments are unlikely to produce useful knowledge, because “it is very hard to learn from very big mistakes.” This is for two reasons: because the scale of such experiments makes it difficult to identify cause and effect, and because discontent will have to be suppressed, as tending to sabotage the plan. This isolates the planner from reliable information about the society for which he is planning, as well as from reasonable criticism (or genuine praise, since such is now compulsory). The scientific character of the enterprise is thereby destroyed.

Anti-naturalist historicists misunderstand physics when they posit fundamental differences between it and sociology. Physicists do not have a priori knowledge of which factors need to be controlled for during an experiment. On the contrary, experiment is the method for discovering such factors. They are therefore mistaken to argue that social experiments must be conducted on the whole of society. Differences between social periods do not imply that they are governed by different laws, as in physics the same laws produce widely variable outcomes.

Popper concluded with the following comments: that progress in the sciences depends on institutions in which freedom of criticism is both possible and encouraged, that such institutions tend strongly toward the production of objective knowledge, and that they are possible only in the context of an open (non-totalitarian) society. In so much as sociology of knowledge posits other sources of knowledge, which is not and never can be objective in any meaningful sense of the word, it is a sham science. The fervent revolutionary rhetoric of historicists betrays a frightened conservatism that seeks to negate change by appealing to imaginary laws of history, whose principal virtue is that they do not change, and whose logical application is to halt change not just in theory, but in fact. Further, the course of history is strongly affected by the progress of knowledge. However, no prediction can reveal the future course that knowledge may take. This in turn precludes knowledge of the future course of history.

This post is the twenty-fifth is a series on the philosophy of history; the previous article in the series is here, the next is here.

Daniel Halverson is a graduate student studying the History of Science and Technology of nineteenth-century Germany. He is also a regular contributor to the PEL Facebook page.

Image of Karl Popper: no known copyright restrictions.