Explainability in recommender systems is both crucial and challenging. Among the state-of-the-art explanation strategies, counterfactual explanation provides intuitive and easily understandable insights into model predictions by illustrating how a small change in the input can lead to a different outcome. Recently, this approach has garnered significant attention, with various studies employing different metrics to evaluate the performance of these explanation methods. In this paper, we investigate the metrics used for evaluating counterfactual explanation methods for recommender systems. Through extensive experiments, we demonstrate that the performance of recommenders, has a direct effect on counterfactual explainers and ignoring it results in inconsistencies in evaluation of explainer methods. Our findings highlight an additional challenge in evaluating counterfactual explainer methods and underscore the need for reporting the recommender performance or considering it in evaluation metrics. All the codes and results are available in our GitHub repository.