1-Simpson’s Paradox: Berkeley Admissions
There has been a revolution in terms of understanding causal inference, launched by Judea Pearl and associates, and based on a graph-theoretic approach. As Pearl, Glymour, and Jewell (2016. Causal Inference: A Primer) state: “More has been learned about causal inference in the last few decades than the sum total of everything that had been learned about it in all prior recorded history. … Yet this excitement remains barely seen among statistics educators, and is essentially absent from textbooks of statistics and econometrics.” Current practice of statistics and econometrics is very much like archaic medical treatments, which inflicted more pain and injury to the patient than the disease. Those who make the effort to learn the theory of causality, can be pioneers of an exciting new approach, which will allow us to distinguish between real relationships and spurious ones.
We begin with a discussion of Simpson’s Paradox, which provides a clear illustration of how and why it is necessary to understand causal linkages, in order to do sensible data analysis. The analysis below is based on real examples. However, we have deliberately changed the numbers to make the calculations easy, and to use identical numbers across several examples. This is to show that dramatically different analyses are required when the unobserved and hidden causal structures are different, even though the actual numerical data remains exactly the same. This point is rarely highlighted in texts, which create the contrary impressions that the data by itself provides us with sufficient information to enable analysis. This illusion is especially created and sharpened by “Big Data” and “Machine Learning” technologies, which appear to inform us that data by itself, in sufficient quantities, can provide us with all necessary information.
The Berkeley Admission Case
Suppose there are two departments Engineering (ENG) and Humanities (HUM), which have differing admissions policies. Due to these policies, 80% of female applicants to ENG are admitted, while only 40% of the female applicants are admitted in HUM. To understand Simpson’s Paradox, it is essential to understand the relation between these departmental admissions rates, and the overall admit rate for females in Berkeley. Assuming, for simplicity, that these are the only two departments, we ask: What is the OVERALL admission rate for female applicants at Berkeley? The answer is that the overall admit ratio is the weighted average of the two admission percentages (80% and 40%). Table 1 show overall admit rate of females with different no. of applicants
Table 1: Overall admit rate for Female applicants
If all females apply to HUM and none to ENG then overall admit rate is 40%. If all females apply to ENG then overall admit rate for females will be 80%. The table shows that the overall admit rate for females can vary from 40% to 80% depending upon proportions of females which apply to the two departments.
Now suppose Berkeley systematically discriminates against males. For male applicants to ENG, the admit ratio is only 60%, much lower than the 80% ratio for females. For male applicants to HUM, the admit ratio is only 20%, much lower than the 40% for females. What will the overall admit rate for males be? As before, this will be a weighted average of the two rates 20% and 60%, where the weights will be the proportion of male applicants to the two departments. The table below shows how the overall admissions ratio varies depending on how many males apply to which department:
Table 2: Overall admit rate for male applicants
The table shows that the overall admit rate for males can vary between 20% and 60% according to how the applicants are distributed between ENG and HUM. We have already seen that overall admit rates for females can vary between 40% and 80%. Now consider the scenario created by the highlighted rows in the table. If 90% of the females apply to HUM, then the female admit ratio will be 44%, close to the 40% admit ratio for females in HUM. If 90% of the males apply to ENG then the admit ratio for males will be 56%, close to the 60% admit ratio for males in ENG. Despite the fact that females are heavily favored in both ENG and in HUM, the overall admit ratio for females (44%) will be much lower than the admit ratio for males (56%). Someone who looks only at the overall admit ratio for males and females will come to the conclusion that Berkeley discriminates against females, which is the opposite of the picture that emerges when looking at departmental admit ratios. This is known as the Simpson’s Paradox.
Interestingly, this is not a hypothetical example. I have simplified the numbers to make the analysis easier to follow, but the actual data for Berkeley admissions follows a similar pattern. The overall admit rates appear to show bias against females. Bickel et. al. (1975) carry out a standard statistical analysis of aggregate admissions data. They test the hypothesis of equality of admit rates for males and females and conclude that males have significantly higher admissions ratio than females. A causal analysis of data attempts to answer the “WHY” question. Why is the admit rate for males higher? To try to learn why the male admit rate was higher, Bickel et. al. (1975) looked at the breakdown by department. Note that the data themselves furnish us with no clue as to what else we need to look at. It is our real world knowledge about colleges, admissions process, departments, which suggests that department-wise analysis might lead to deeper insights. This shows how real world knowledge, which goes beyond the data, matters for data analysis. Doing the analysis on the departmental level leads to an unexpected finding — each department discriminates in favor of women. Philosophers call this “counter-phenomenal”. The phenomena — the observation — at the aggregate level suggests that Berkeley discriminates against women. But a deeper probe into reality reveals that the opposite is true. This shows the necessity of going beyond the surface appearances, the observations, to deeper structures of reality, in order to understand the phenomena. This is in conflict with Kantian and Empiricist ideas that observations by themselves are sufficient, and we do not need to probe deeper.
When we discover a conflict between the phenomena and our exploration of the noumena — the deeper and hidden structures of reality — then we are faced with the necessity of explaining this conflict. Because both departments discriminate against males, the explanation that Berkeley admissions process discriminates against females is no longer acceptable. Bickel et. al. (1975) do the data analysis and come up with the deeper explanation. ENG is easier to get into, and HUM is more difficult. Females choose to apply to the more difficult department and hence end up with lower admit ratios. Males choose to apply to the easier department, and hence have higher admit ratios. The search for causal explanations does not stop here. We can then ask: WHY do females choose humanities? We can also ask: WHY is ENG easier to get into, and WHY is HUM more difficult to get into? For both of these questions, there are several possible hypotheses which could be true, and which could be explored using data or qualitative techniques. In the next section, we will consider some other causal structures for admissions, which lead to radically different answers to the WHY questions, even though the observed data remains exactly the same.
For full article, see: https://bit.ly/simp7