Productivity losses in the manufacturing industry have to be minimized to perform at total capacity consistently. If productivity losses are neglected, a reduction in overall revenue results from an underperforming manufacturing plant. Deep Learning (DL) and Computer Vision (CV) can serve as an automated surveillance system for continuous monitoring. The combination of DL and CV, which act as eyes and brain, can capture the activities in the manufacturing plant and convert such visual data into meaningful information in the form of object detections. The requirement is to detect the status of maintenance, operator and machines and their interactions based on user-defined thresholds. Metrics such as machine utilization, Mean-Time to Repair, Mean-Time to Failure, Operator in position and maintenance behavior are derived from these detections. This paper compares two approaches using data for a real-time analytics dashboard to track productivity losses in a manufacturing setup. The results are compared to select a better performing approach regarding their contribution to productivity analytics accuracy. The first approach uses weights of the Yolov4 model to detect maintenance and operator as a person in combination with HSV colour filtering for machine status and maintenance operator classification. Later uses Transfer learning on Yolov4 to detect machine status, maintenance and operator. Literature shows that transfer learning is significantly faster and consumes less data than conventional model training, so an approach to compare conventionally trained models was unnecessary. The results from the experiment show that the second approach using transfer learning was accurate compared to the first approach.