Going beyond accuracy in the evaluation of a recommender system is an aspect that is receiving more and more attention. Among the many perspectives that can be considered, the impact of presentation bias is of central importance. Under presentation bias, the attention of the users to the items in a recommendation list changes, thus affecting their possibility to be considered and the effectiveness of a model. Page-wise within-subject studies are widely employed in the recommender systems literature to compare algorithms by displaying their results in parallel. However, no study has ever been performed to assess the impact of presentation bias in this context. In this paper, we characterize how presentation bias affects different layout options, which present the results in column- or row-wise fashion. Concretely, we present a user study where six layout variants are proposed to the users in a page-wise within-subject setting, so as to evaluate their perception of the displayed recommendations. Results show that presentation bias impacts users clicking behavior (low-level feedback), but not so much the perceived performance of a recommender system (high-level feedback). Source codes and raw results are available at https://tinyurl.com/PresBiasSIGIR2023.

Rows or Columns? Minimizing Presentation Bias When Comparing Multiple Recommender Systems

Boratto L.
2023-01-01

Abstract

Going beyond accuracy in the evaluation of a recommender system is an aspect that is receiving more and more attention. Among the many perspectives that can be considered, the impact of presentation bias is of central importance. Under presentation bias, the attention of the users to the items in a recommendation list changes, thus affecting their possibility to be considered and the effectiveness of a model. Page-wise within-subject studies are widely employed in the recommender systems literature to compare algorithms by displaying their results in parallel. However, no study has ever been performed to assess the impact of presentation bias in this context. In this paper, we characterize how presentation bias affects different layout options, which present the results in column- or row-wise fashion. Concretely, we present a user study where six layout variants are proposed to the users in a page-wise within-subject setting, so as to evaluate their perception of the displayed recommendations. Results show that presentation bias impacts users clicking behavior (low-level feedback), but not so much the perceived performance of a recommender system (high-level feedback). Source codes and raw results are available at https://tinyurl.com/PresBiasSIGIR2023.
2023
9781450394086
Recommender systems; user study design; presentation bias
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11584/390355
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 1
  • ???jsp.display-item.citation.isi??? 1
social impact