Peter joshi

Every Data Scientist needs to know these ideas. They will blow your mind. 1. Correlation vs Causation P(A | B) is the probability of A given B. It is the probability that we will observe A given that we have already observed B. P(A | do(B)) is the probability of A given do(B). It is the probability that we will observe A given that we have intervened to cause B to happen. In this context, an intervention simply means to take an action of some kind. Therefore do(B) means to take an action which causes B to happen. The expressions P(A | B) and P(A | do(B)) might seem very similar but they represent very different situations. 2. We can only learn P(A|B) from the data alone. Bob has an extremely accurate weather app and is always very good about bringing his umbrella when it rains. We observe Bob over several years and we find that whenever it rains, Bob always has his umbrella and he never brings his umbrellas on days when it doesn't rain. In the language of probability, we say P(Umbrella | Rain) = 1 and P(Rain | Umbrella) = 1 as well. What we can learn from this data alone is how to predict whether it rains with a 100% accuracy by checking whether Bob has an umbrella. We can also learn to predict with 100% accuracy whether Bob has an umbrella by checking if it's going to rain. What we cannot learn is what will happen if we give Bob an umbrella on a random day of our choosing. The answer to this question is P(Rain | do(Umbrella) ) and it's unknowable from the data alone. We need prior knowledge about how the world works to properly interpret the data we collected. We need to know that rain has an effect on Bob's behavior, but Bob's behavior has no effect on the rain. Information about the effects of interventions are simply not available in raw data unless it is collected by controlled experimental manipulation. 3. Scientific Experiments work because they produce a very special kind of data. You may have heard of what many people call a scientific experiment. Take a collection of objects, animals or people. Randomly split that collection into a control group and a treatment group. Apply your intervention to the treatment group while leaving the control group alone. If you observe any differences between the treatment group and the control group, it is logical to attribute these differences to the treatment. You can therefore say the differences were caused by the treatment. In statistics, the procedure I just described is called a Randomized Controlled Trial. It is a procedure for generating a specific kind of data where: P(Difference | Treatment) = P(Difference | do(Treatment) ) This is why traditional science experiments work. They are designed to capture causal information. This is not the case for vast majority of data that we collect in society. Without human guidance or access to real world knowledge, statistical algorithms and artificial intelligences can only learn P(A | B) from the raw data. This is a fundamental mathematical limitation on the use of data alone. That's it for now. This post is part of a series of posts about the concept of causal inference. They are based on the content of the Book of Why by Judea Pearl with lots of commentary from me. Follow me (@kareem_carr) so you don't miss out on the next post. Please show support by liking and retweeting the thread.

What is Causal Inference? Causal Inference is a new science of causation. This field is nothing less than a revolution in how scientists understand data. Read on to learn more. This is the first post in a series based on the Book of Why by Judea Pearl. I will be reading the book and sharing the big insights with my followers. When I first started learning causal inference, I didn't have a clear idea of the problems that casual inference was trying to solve. My misconceptions made it harder to understand the material than it would have been otherwise. So, before we get into the ideas of the book, I want to help you avoid these common misconceptions. 1. Causal Inference is NOT just regular science All sciences strive to infer causes within their domain of expertise. Therefore, it might not be obvious to you what makes casual inference any different. This is the reason why I sometimes call this new field mathematical causal inference. This term emphasizes that what sets causal inference apart is the mathematical framework it uses to describe causation. 2. Causal Inference is NOT directly about inferring causes Based on the name, new learners often think causal inference is solving the following problem: Given a list of candidate variables, how can we select the ones that have a real causal effect on our outcome of interest? This is not what causal inference does. Causal inference is solving a different problem: Assuming our beliefs about the causal relationships between all the variables is accurate, what is the best estimate of the causal relationship between a particular candidate variable and the outcome of interest? Very roughly speaking, causal inference tells us whether based on our causal beliefs, the association between two variables is bigger or smaller than their true casual relationship. 3. The Example of Alice and Bob Alice thinks genes strongly affect addictive behaviors like smoking. She also thinks genes have an effect on who gets cancer. Bob agrees that genes very likely have an effect on cancer, but Bob thinks complicated social behaviors like addiction are completely due to social factors, not genes. Causal inference allows us to evaluate the same data according to both Alice's and Bob's beliefs about the underlying causal relationships. This allows for various outcomes: 1. Avoiding unnecessary arguments. If Alice and Bob get very similar estimates for the causal relationship between genes and cancer, this implies that the disagreement about the relationship between genes and behavior is not that important. This allows scientists to move forward by focusing on the factors that really matter. 2. Agreeing to disagree. If the difference in estimates of the casual relationship between genes and cancer is large, causal inference allows both Alice and Bob to continue to explore the same data according to their very different assumptions about the causal relationships. This gives scientists and policy makers autonomy to pursue different interpretations of the same data. 4. Casual Inference builds doesn't replace statistics. It makes it more powerful. Causal inference allows us to adjust our statistical estimates of the strength of particular casual relationships based on our beliefs about the casual relationships between the variables. This is why some experts in causal inference (like the epidemiologist @epiellie) prefer to use the term causal effect estimation to refer to the field causal inference. That's it for now. My next post (coming soon!) will explore how causal inference creates a mathematical model of causation and what makes this approach so special. (You can find these posts using the hashtag #KareemReads) Follow me (@kareem_carr) for more content like this. If you want to show support, like and retweet the thread.