Rebuilding Statistics on Non-Positivist Foundations

Flawed Positivist Foundations of Statistics

Recently, I finished a set of 7 lectures based on a draft of my new textbook on “Real Statistics: A Radical Approach”. All of these lectures explained why modern statistics is built on wrong foundations — so wrong that repairs are not possible. Instead, we must abandon the entire structure developed over the past century, and rebuild the discipline on new foundations. Below, I will provide an outline, summary, and links to the chapters covered. My goal is to writeup a book based on this draft. I invite people with the relevant skill set to help him with this process of converting the rough draft into a book.

The fundamental problem lies in epistemology — the theory of knowledge — which underlies modern statistics. The discipline was created in the early twentieth century under the strong influence of logical positivism. This philosophy asserts that knowledge is only based on observations and logic. Positivism had a spectacular collapse later, when it was discovered that scientific theories make essential use of unobservables, and cannot be reduced to ideas involving observables only. The discipline of statistics (and economics) has been built on foundations of positivism. Collapse of positivism makes it necessary to rethink these foundations, and rebuild, but this has not been done. Both disciplines continue to use the same methodology which has been proven to be fatally flawed by philosophers. This book aims to explain these issues in greater detail

Very briefly, logical positivism asserts that domain of knowledge should be confined to those statements which can be verified by observations and logic. Then, who made the statement, and why he/she made it, are not relevant to the assessment of it’s validity. In Islamic epistemology, we look at the reliability of the person making the statement first, before evaluating the statement. The fundamental claim I am making appears outrageous: the entire structure of statistical theory, as it has developed over the past century, is flawed beyond repair, and must be abandoned. In accordance with Islamic epistemological principles, I thought it best to begin by explaining my life experiences which led me to these radical views. My hope is to establish credibility, before we start the discussion of statistics. Accordingly, my draft book begins with a discussion of my life-experiences which led me to the creation of this textbook. The video is not directly related, and discusses my education at MIT as a teenager, from my current perspective as a senior citizen:

Chapter 0: Life Journey:

1. Beginnings: I describe how a Western education led me to the conviction that the knowledge created by the West over the past three centuries is the most magnificent achievement of mankind. Later, my experience with Tableegh led me to the heart-certainty that the final message of God to mankind, embodied in the Quran and Sunnah, provides complete and perfect guidance for all times. I could not find any way to reconcile these two diametrically opposite ideas, creating a dramatic conflict between my head and heart.
2. Dreams: Reading Edward Said’s powerful and influential book “Orientalism”, which destroyed an entire category of thought, planted the seeds of an idea about how this conflict could be resolved. What if the entire discipline of Economics, in all its sophistication and depth, is nothing more than a projection of European power? I realized that it would take decades to explore this idea in depth, and find solid evidence. I decided to embark on this journey of a thousand miles, by taking small steps, and gathering evidence of small problems in limited subdomains. The hope was to collect sufficient ammo, bit by bit, to launch an attack on the citadel of Western knowledge.
3. Europeans Reject the Unseen: I started to find chinks in the armor of the formidable rigorous mathematical structures of economics. I learned from Paul Bairoch: Economics and World History: Myths and Paradoxes that history does not support one of the favorite ideas of economists — that free trade is good for everyone. Similarly, the foundation of microeconomics, that every person tries to maximize pleasure, does not correspond to actual behavior. Tracing these anomalies to their roots led me to the discovery that the fundamental problem was a rejection of the unobservables as a source of knowledge. In turn, this resulted from European historical experience with religious wars lasting for more than a century. This led to a rejection of religion, which was extended to a rejection of all unobservables, as a basis for knowledge.
4. The Poison of Positivism: Having rejected Christianity as a basis for knowledge, European had to search for new foundations of knowledge. The Enlightenment Project was based on the idea that the entire stock of human knowledge must be built up from observations and logic, and include only certainties, without any element of doubt. In early 20th Century, this idea became dominant in the form of logical positivism. Since only empirically demonstrable statements counted as truth, all moral statements, character building, community, compassion were ruled out of the bounds of knowledge. This had a dramatic effect on University education, which shifted from character building to purely technical education. The attempt to achieve certainty — which is impossible to achieve for fallible human beings — led to the cloaking of toxic moral foundations as self-evident truths.
5. Impact on Statistics: Logical Positivist theories of knowledge were used to build the foundations of statistics in the early 20th Century. Two critical ideas which are fundamental to statistics are unobservable. Causation refers to a hidden structure of reality such that when event E occurs then event F necessarily follows. We can observe that one event follows the other, but it is impossible to observe that this is necessarily so. That a cause which was effective in the past will continue to be effective in the future. The observables can only provide us with correlations, and modern statistics still does not have a satisfactory theory of causation. Similarly, probability concerns what might have happened. This is also unobservable. The positivist foundations of statistics have prevented the development of satisfactory treatments of probability and causality.