Knowledge distillation is a model compression technique that facilitates the transfer of knowledge from a large teacher model to a lightweight student model, thereby enabling the efficient deployment of high-performance deep learning models in resource-constrained environments such as mobile devices and edge computing. Although traditional knowledge distillation employing a single teacher model has proven effective, utilizing multiple teacher models may further enhance the performance of the student model by integrating complementary knowledge derived from diverse architectures and training experiences. This study systematically examined the impact of varying the number of teacher models (one, two, or three) on the performance of knowledge distillation and explored the balance between the optimal number of teacher models and computational efficiency. The experiments were conducted using the CIFAR-10 dataset. The teacher models employed were ShuffleNet V2 1.0x (for the single-teacher scenario), MobileNet V2 (added for the two-teacher scenario), and ResNet18 (added for the three teacher scenario). A lightweight ShuffleNet V2 0.5x was used as the student model. Data preprocessing included horizontal flipping, random cropping and normalization. Training was performed using SGD optimization with a learning rate of 0.1, momentum of 0.9, batch size of 128, and 30 epochs. Knowledge distillation employs a loss function that combines the hard label loss and soft teacher output loss. In the multi-teacher setting, the average outputs of the teacher models were used to generate the integrated knowledge. The implementation was performed using the PyTorch framework with GPU-acceleration. The experimental results indicated that the single-teacher model achieved an accuracy of 51.8%with an unstable learning curve and a tendency for early convergence. When two teacher models (ShuffleNet V2 1.0x and MobileNet V2) were used, the accuracy improved to 63.8%, and learning stability was enhanced. With three teacher models (including ResNet18), the highest accuracy of 89.5% was recorded, demonstrating the most stable and consistent performance improvement. Compared with a single teacher, the multi-teacher setup optimized the loss function more effectively and exhibited smoother convergence during training. This suggests that multi-teacher ensembles provide knowledge from diverse perspectives, thereby strengthening the generalization capability of the student model.

Keywords

Knowledge Distillation, CIFAR-10, Multiple Teachers Models,  PyTorch and SGD