Despite the growing interest in explainable AI in the RecSys community, the evaluation of explanations is still an open research topic. Typically, explanations are evaluated using offline metrics, with a case study, or through a user study. In my research, I will have a closer look at the evaluation of the effects of explanations on users. I investigate two possible factors that can impact the effects reported in recent publications, namely the explanation design and content as well as the users themselves. I further address the problem of determining promising explanations for an application scenario from a seemingly endless pool of options. Lastly, I propose a user study to close some of the research gaps established in the surveys and investigate how recommender systems explanations impact the understanding of users with different backgrounds.