Amidst the epoch of the fourth industrial revolution, a discernible imperative surface within the steel industry necessitating the replacement of the archaic Defect Inspection System (DIS). The profound fiscal repercussions stemming from substandard steel underscore the exigency for this transition. Real-time diagnostics, a pivotal facet of quality control in manufacturing, grapple with inherent challenges, notably low automation and inconsistent flaw detection on steel surfaces. In response, a groundbreaking approach has materialized in the form of machine vision-based models, strategically devised to surpass the capabilities of conventional DIS and elevate the quality of produced steel. In the course of our study, we addressed flaws in six hot-rolled steel predicaments, leveraging a dataset encompassing ten critical surface defects: inclusion, pitted surface, crazing, rolled-in scale, patches, and scratches, thereby confronting the challenges previously articulated. Upon meticulous analysis of the dataset, our model, the Vision-Based Transformer (VIT), attained an exceptional accuracy rate of 98%. Four distinct machine learning models—Xception, ResNet50V2, EfficientNetB2, and MobileNetV2—were enlisted for performance evaluation, ultimately revealing the superiority of the VIT in the domain of vision-based Defect Inspection Systems