Online Advertising Incrementality Testing: Practical Lessons And Emerging Challenges

Tutorial Website https://joel-barajas.github.io/cikm2021-incrementality-testing/

Tutorial Description

Online advertising has historically been approached as an ad-to-user matchingproblem within sophisticated optimization algorithms. As the research andad-tech industries have progressed, advertisers have increasingly emphasizedthe causal effect estimation of their ads (incrementality) using controlledexperiments (A/B testing).

With low lift effects and sparse conversion, the development of incrementalitytesting platforms at scale suggests tremendous engineering challenges inmeasurement precision. Similarly, the correct interpretation of results addressinga business goal requires significant data science and experimentation researchexpertise.We propose a practical tutorial in the incrementality testing landscape, including:

  • The business need
  • Literature solutions and industry practices
  • Designs in the development of testing platforms
  • The testing cycle, case studies, and recommendations

We provide first-hand lessons based on the development of such a platform in amajor combined DSP and ad network, and after running several tests for up totwo months each over recent years.

We elaborate more on the user privacy implications in online experimentationand incrementality testing. We aim to motivate the research community to focuson solutions under these emerging constraints.

Tutorial Organisers

  • Head shot of Joel Barajas
    Joel Barajas
    Yahoo Research, Verizon Media
    Joel has over 11 years of experience in the online advertising industry with research contributions at the intersection of Ad tech, Marketing Science, and Experimentation. He has experience with Ad load personalization and experimentation in a publisher marketplace. Within Marketing Data Science orgs, he has supported regular budget allocation and Media Mix Models in multi-channel advertising. With a PhD dissertation focussed on ad incrementality testing, his published work has appeared in top outlets including INFORMS Marketing Science Journal, ACM CIKM, ACM WWW, SIAM SDM. He led the science development and marketing analytics of the incrementality testing platform in a multidisciplinary team. He currently oversees most incrementality tests at yahoo! ad network and DSP. Joel also leads the science development in CTV and linear TV measurement modeling. He holds a B.S. (with honors) in Electrical and Electronics Engineering from the Tecnológico de Monterrey, and a PhD in Electrical Engineering (with emphasis on statistics) from UC Santa Cruz.
  • Head shot of Narayan Bhamidipati
    Narayan Bhamidipati
    Yahoo Research, Verizon Media
    Narayan has over 14 years of experience in Computational Advertising and Machine Learning. He currently leads a team of researchers focused on providing state-of-the-art ad targeting solutions to help ads be more effective and relevant. This includes creating various contextual targeting products to reduce the company's reliance on user profiles and help improve monetization in a more privacy aware world. Alongside that, Narayan ensures that the user profile based ad targeting products continue to improve despite the decline of tracking data. In addition, Narayan is keen on developing the most accurate ad effectiveness measurement platform which would help the company attract more revenue by proving the true value of the ad spend on our platforms. He holds B.Stat(Hons), M.Stat and PhD(CS) degrees, all from the Indian Statistical Institute, Kolkata.
  • Head shot of James G. Shanahan
    James G. Shanahan
    Church and Duncan Group Inc and UC Berkeley
    Dr. James G. Shanahan has spent the past 30 years developing and researching cutting-edge artificial intelligence systems, splitting his time between industry and academia. For the academic year 2019-2020, Jimi held the position of Rowe Professor of Data Science at Bryant University, Rhode Island. He has (co) founded several companies that leverage AI/machine learning/deep learning/computer vision in verticals such as digital advertising, web search, local search, and smart cameras. Previously he has held appointments at AT&T (Executive Director of Research), NativeX (SVP of data science), Xerox Research (staff research scientist), and Mitsubishi. He is on the board of Anvia, and he also advises several high-tech startups including Aylien, ChartBoost, DigitalBank, LucidWorks, and others. Dr. Shanahan received his PhD in engineering mathematics and computer vision from the University of Bristol, U. K. Jimi has been involved with KDD since 2004 as an author, as a tutorial presenter, and as a workshop co-chair; he has actively been involved as a PC/SPC member over the years also.

Tutorial Abstract

Online advertising has historically been approached as an ad-to-user matching problem within sophisticated optimization algorithms. As the research and ad-tech industries have progressed, advertisers have increasingly emphasized the causal effect estimation of their ads (incrementality) using controlled experiments (A/B testing). With low lift effects and sparse conversion, the development of incrementality testing platforms at scale suggests tremendous engineering challenges in measurement precision. Similarly, the correct interpretation of results addressing a business goal requires significant data science and experimentation research expertise. We propose a practical tutorial in the incrementality testing landscape, including: item The business need Literature solutions and industry practices Designs in the development of testing platforms The testing cycle, case studies, and recommendations. We provide first-hand lessons based on the development of such a platform in a major combined DSP and ad network, and after running several tests for up to two months each over recent years.