The time study approach is commonly performed to eval¬uate the personnel utilization for effective resource management. It requires high engineering effort and is a time-consuming process. The results are often poorly represented due to the small sample size and averaging approach. In this study, a vision-based operator activity recognition sys¬tem is developed. It comprises of four modules, mainly the human detection and tracking, color detection, color light detection and sift detection. First, the key operator activities are defined. Then, the key elements to describe the activity are identified. Their features are then learned from a set of reference frames. A set of input frames from a video stream will be processed and the key elements of an activity can be matched and tracked in the new frames. The results provide the change in states and locations of those key elements. Subsequently, expert rules are defined to determine the activity initialization and completion. The overall system illustrates the ability to recognize four key activities: Transactional documentation; Input tray loading; Reel change; Tape carrier change. The frequency of activity occurrence and time duration is determined via the defined expert rules. A more accurate and comprehen¬sive evaluation of the operator utilization is obtained with minimal engineering effort.