The non-contact inspection of surface defects has become more and more important in the manufacturing industrial systems because of the great demands on the quality of high surface finishes. The machine learning achieved impressive recognition rates in image classification tasks. In order to exploit those capabilities, this paper represents a detection and classification of surface defects on hot rolled steel strip by means of differently captured digital intensity images of that process samples. The feed-forward artificial neural networks and deep auto-encoder network as classifiers are trained for detecting six popular classes of steel defects, i.e., crazing, patches, pitted surface, inclusion, rolled-in scale, and scratches. A comparative study between the two classifiers is done with respect to the cross validation and the confusion matrix. The results demonstrate excellent defect detection outcome with very low false rates.