At online supermarket Picnic, we implemented both traditional collaborative filtering and a hybrid method to provide recipe recommendations at scale. This case study presents findings from the online evaluation of these algorithms, focusing on the repeat-explore trade-off. Our findings allow other online retailers to gain insights into the importance of thoughtful model design in navigating this important balance. We argue that even when exploiting known preferences proves highly beneficial in the short term, prioritizing exploratory content is essential for long-term customer satisfaction and sustained growth. Our research lays the groundwork for a compelling discussion on defining success in balancing the familiar and the novel in online grocery shopping.