Over the recent years, Deep Reinforcement Learning (DRL) has been an essential practice in enhancing profitability in healthcare sector to deliver customized interventions in hospitals, optimize resources, and make predictive models, especially in emerging economies. Nevertheless progress, challenges such as model interpretability and real-world integration remain.  This study covers optimization approaches in DRL, with a focus on Proximal Policy Optimization (PPO) for its stability, efficiency, and application in healthcare. Beyond Proximal Policy Optimization (PPO), we will go into a larger repertoire of algorithms such as Soft Actor-Critic (SAC) and Twin Delayed Deep Deterministic Policy Gradient (TD3) and Deep Q-Networks (DQN) and Multi-Agent DRL (MADRL). We critique their use in various fields of healthcare that include adaptive radiation therapy, chronic illness treatment, and detection of diseases at a premature stage. As part of our methodology, we performed a broadened systematic review of 2004-2024 with the combination of grey literature and varying databases. As indicated in the evidence, DRL has a great potential in improving the outcomes of patients and in efficiency of care. Future research directions include emerging frontiers in Explainable DRL, Federated DRL, and Digital Twin-based interventions as one of their future directions. The paper tries to advice on how DRL can be adopted in healthcare systems with a goal to achieve patient-centered, efficient, and data-driven clinics decision support.