Convolutional Neural Network (CNN) is used for processing image data in various applications, including logistics systems, production, etc. This research proposes a method for optimizing the structure of CNN. Many CNN models already use various structures, but there is no certain well-known framework for designing high-accuracy CNN models yet. Unlike previous studies, this study identifies combinations of various types and number of layers in CNN. Considering a limited number of alternative layer combinations when designing CNN models would reduce the computation time of the model generation (and increase the performance of the CNN model because it can consider more design alternatives during the model generation phase, given the available time). This study (1) extracts the combination of layers from many state-of-the-art CNN models that ensure the best information is obtained and (2) proposes a module-based framework to optimize the CNN structure. The numerical experiment shows that the proposed framework produces better accuracy than the compared best-known models. This research will provide insights on good combinations of layers and help researchers develop new CNN structures with high accuracy. Convolutional Neural Network (CNN) is used for processing image data in various applications, including logistics systems, production, etc. This research proposes a method for optimizing the structure of CNN. Many CNN models already use various structures, but there is no certain well-known framework for designing high-accuracy CNN models yet. Unlike previous studies, this study identifies combinations of various types and number of layers in CNN. Considering a limited number of alternative layer combinations when designing CNN models would reduce the computation time of the model generation (and increase the performance of the CNN model because it can consider more design alternatives during the model generation phase, given the available time). This study (1) extracts the combination of layers from many state-of-the-art CNN models that ensure the best information is obtained and (2) proposes a module-based framework to optimize the CNN structure. The numerical experiment shows that the proposed framework produces better accuracy than the compared best-known models. This research will provide insights on good combinations of layers and help researchers develop new CNN structures with high accuracy.