Remote Access Trojans (RATs) pose a significant and persistent threat to Android security, enabling cybercriminals to gain unauthorized access to sensitive user data and control infected devices. Despite advancements in cybersecurity, detecting Android RATs remains an open challenge due to their evolving evasion techniques and stealthy behavior. Traditional detection methods often fail to keep pace with the rapid evolution of these threats, hence, there is a need for more robust and intelligent detection mechanisms.

In this thesis, we propose an advanced detection framework leveraging ensemble machine learning models to effectively identify and classify Android RATs. By analyzing a diverse dataset of malicious Android applications, this study examines dynamic features such as system calls and static features like permissions, intent filters and metadata to develop a comprehensive detection approach for Android RATs. The experimental results demonstrate that XGBoost model achieves the best trade-off between precision, recall, and false positive rate, making it the most reliable model for RAT detection. Furthermore, this research incorporates optimized feature selection techniques to enhance model performance and improve detection accuracy. Additionally, we investigate the effectiveness of different Large Language Models (LLMs) in identifying Android RATs. This work provides valuable insights into the strengths and limitations of current machine learning-based approaches in detecting Android RATs by benchmarking multiple models and analyzing their comparative effectiveness. The proposed detection framework aims to enhance mobile security by equipping Android users and cybersecurity professionals with a more accurate, efficient, and adaptive solution against evolving RAT threats.