Text

Novelty and Primacy : A Long-Term Estimator for Online Experiments

Online experiments are the gold standard for evaluating impact on user experience and accelerating innovation in software. However, since experiments are typically limited in duration, observed treatment effects are not always stable, sometimes revealing increasing or decreasing patterns over time. There are multiple causes for a treatment effect to change over time. In this article, we focus on a particular cause, user-learning, which is primarily associated with novelty or primacy. Novelty describes the desire to use new technology that tends to diminish over time. Primacy describes the growing engagement with technology as a result of adoption of the innovation. Estimating user-learning is critical because it holds experimentation responsible for trustworthiness, empowers organizations to make better decisions by providing a long-term view of expected impact, and prevents user dissatisfaction. In this article, we propose an observational approach, based on difference-in-differences technique to estimate user-learning at scale. We use this approach to test and estimate user-learning in many experiments at Microsoft. We compare our approach with the existing experimental method to show its benefits in terms of ease of use and higher statistical power, and to discuss its limitation in presence of other forms of treatment interaction with time.

Ketersediaan

Informasi Detail

Judul Seri

TECHNOMETRICS; Vol.64 No.4 November 2022

No. Panggil

-

Penerbit

: .,

Deskripsi Fisik

p. 524-534

Bahasa

English

ISBN/ISSN

-

Klasifikasi

NONE

Tipe Isi

-

Tipe Media

-

Tipe Pembawa

-

Edisi

-

Subjek

DIFFERENCE-IN-DIFFERENCES
A/B TESTING
TRUSTWORTHINESS
STATISTICAL POWER
USER-LEARNING

Info Detail Spesifik

https://doi.org/10.1080/00401706.2022.2124309

Pernyataan Tanggungjawab

Soheil Sadeghi, Somit Gupta, Stefan Gramatovici, Jiannan Lu, Hao Ai

Versi lain/terkait

Lampiran Berkas

Komentar