This paper focuses on a comparison study between the traditional deep learning techniques like LSTMs and GRUs with the latest state-of-the-art transformers for the task of classifying a comment based on its toxicity. Specifically, four different deep learning models are built, implemented and trained on a standard dataset for comparing the performance of the models. The models that are explored and compared are Bi-directional LSTMs, GRUs, Bi-directional LSTMs with CNNs and Transformers (with pre-trained RoBERTa weights).