Live Session
Thursday Posters
Research
It's (not) all about that CTR: A Multi-Stakeholder Perspective on News Recommender Metrics
Hanne Vandenbroucke (imec-SMIT Vrije Universiteit Brussel) and Annelien Smets (imec-SMIT Vrije Universiteit Brussel)
Abstract
Recommendation systems are increasingly adopted by news media organizations. Existing literature examines various aspects of news recommendation systems (NRS) from a computational, user-centric, or normative perspective. Yet, scholars have argued to study the complexity of real-world applications surrounding the development and evaluation of NRS. More specifically, understand why different stakeholders within the organization prioritize different values for developing NRS and how those values are operationalized into metrics. Recently, a multi-stakeholder approach to NRS has been adopted, allowing to capture this complexity more comprehensively. However, limited research has focused on the distinct key performance indicators (KPIs) and metrics considered valuable by different stakeholders in evaluating NRS.Based on 11 interviews conducted with different professionals from two commercial media organizations in Flanders and The Netherlands, this paper shows that stakeholders prioritize different KPIs and metrics related to the reach-engagement-conversion-retention funnel. The evaluation of NRS performance is often limited to short-term metrics like CTR, overlooking the multiplicity of stakeholders involved. Our findings reveal how different purposes, KPIs, and metrics are valued from the journalistic, commercial, and tech logic. In doing so, this paper contributes to the multi-stakeholder approach to NRS, advancing our understanding of the real-world complexity of NRS development and evaluation.