Live Session
Session 12: Optimisation and Evaluation 2
Reproducibility
Informfully - Research Platform for Reproducible User Studies
Lucien Heitz (University of Zurich), Julian A. Croci (University of Zurich), Madhav Sachdeva (University of Zurich) and Abraham Bernstein (University of Zurich)
Abstract
This paper presents Informfully, a research platform for content distribution and user studies. Informfully allows to push algorithmically curated text, image, audio, and video content to users and automatically generates a detailed log of their consumption history. As such, it serves as an open-source platform for conducting user experiments to investigate the impact of item recommendations on users’ consumption behavior. The platform was designed to accommodate different experiment types through versatility, ease of use, and scalability. It features three core components: 1) a front end for displaying and interacting with recommended items, 2) a back end for researchers to create and maintain user experiments, and 3) a simple JSON-based exchange format for ranked item recommendations to interface with third-party frameworks. We provide a system overview and outline the three core components of the platform. A sample workflow is shown for conducting field studies incorporating multiple user groups, personalizing recommendations, and measuring the effect of algorithms on user engagement. We present evidence for the versatility, ease of use, and scalability of Informfully by showcasing previous studies that used our platform.