The healthcare industry is strained and would benefit from personalized treatment plans for treating various health conditions (e.g., HIV and diabetes). Reinforcement Learning is a promising approach to learning such sequential recommendation systems. However, applying reinforcement learning in the medical domain is challenging due to the lack of adequate evaluation metrics, partial observability, and the inability to explore due to safety concerns. In this line of work, we identify three research directions to improve the applicability of treatment plans learned using offline reinforcement learning.