How do we mitigate 'Selection Bias' in our A/B testing for product features?

All answers to this question.

Selection bias is a silent killer of product analytics. If your test group isn't randomly assigned, your p-values are essentially meaningless. "Propensity Score Matching" (PSM) is a great post-hoc way to fix this. It allows you to create a "synthetic" control group by matching your beta users with non-beta users who have similar characteristics (age, tenure, activity level). However, the better approach is "Randomized Controlled Trials" (RCT) from the start. Use a server-side flag to randomly assign users to Group A or B without them knowing, which eliminates the "opt-in" bias entirely and gives you a much cleaner signal of true feature impact.

   Answered 2025-09-24 by Margaret Moore

If we use server-side randomization, how do we handle the "Network Effect" where users in Group A might talk to users in Group B and influence their behavior?

   Answered 2025-09-26 by Brian Allen

• That's known as "Interference" or "SUTVA violation," Brian. In social or collaborative apps, you should use "Cluster Randomization." Instead of randomizing by user, you randomize by entire geographical regions or distinct networks. This ensures that users who interact with each other are all in the same bucket. It requires a much larger sample size to reach statistical significance, but it’s the only way to get an accurate reading when user actions are interdependent and could potentially contaminate the control group results.

     Commented 2025-09-28 by Kevin Scott

Always check your "Sample Ratio Mismatch" (SRM). If you expect a 50/50 split and get 55/45, something is wrong with your randomization engine.

   Answered 2025-09-30 by Nancy King

• Exactly, Nancy. SRM is the first thing I check. Thomas, if you haven't looked at your SRM yet, do that before you dive into complex stuff like Propensity Matching.

     Commented 2025-10-02 by Thomas Anderson