Description:

A/B testing is widely used to compare two alternatives of doing something in order to find out the better alternative. Typical A/B testing involves statistical hypothesis testing which is not intuitive. On the other hand, Bayesian methods are much more intuitive and are based on less assumptions. This talk aims to give a brief on how to do an A/B test with Bayesian methods using Python.

Talk outline

• What is A/B testing?
  • Objective of an A/B test
  • Where is it used?
  • A couple of real life examples
  • Exploration-exploitation tradeoff
• An example of website conversion rate
  • definition of conversion rate and examples
  • observed vs real conversion rate (CR)
• Why not just compare CRs directly?
  • Simulation - effect of sample size on observed CR for the same underlying real CR (uses numpy, scipy)
  • same observed CR does not mean same real CR
  • Need to deal with uncertainty introduced by finite sample
• Two approches
  • Frequentist
  • Bayesian
• Frequentist approach
  • assumptions
  • actual calculation of p-value using scipy
• Limitations of frequentist approach
  • complex and not so intuitive
  • arbitrary cut-off for p-value (0.05)
  • p-value can vary a lot during the test - a simulation
• Bayesian approach
  • brief intro to Bayes theorem and Bayesian method
  • how does it deal with uncertainty
  • working of same example using Bayesian approach
  • using Thompson sampling for better A/B tests
  • computations using numpy, scipy, plotting with matplotlib
  • a short note about pymc3 for advanced bayesian models (if time permits)

Prerequisites:

• Basic understanding of Python
• Basic understanding of probability

Speaker Info:

Vaibhav Pawar is the head of analytics at Loylty Rewardz. He has 10+ years of experience in using data science to solve business problems across industries like banking, retail, airlines, insurance etc. in the areas of marketing, product, digital and consumer loyalty analytics. He has deep understanding of machine learning techniques and has hands-on experience in automating and deploying ML solutions at scale. He graduated from IIT Bombay in 2008. His master's thesis was in the field of Bayesian networks.

Speaker Links:

https://www.linkedin.com/in/vaibhav-pawar-588391a/