3rd Asia Pacific International Conference on Industrial Engineering and Operations Management

Toward Optimum Transformer Model for Sequence-to-Sequence Data Transformation under Low-resource Computation Constraint

Yaya Heryadi, Cuk Tho, Bambang Dwi Wijanarko, Dina Fitria Murad & Kiyota Hashimoto
Publisher: IEOM Society International
0 Paper Citations
1 Views
1 Downloads
Track: Computers and Computing
Abstract

Accurate language translator applications running on low-resource computing devices such as smartphone is very instrumental to support tourism industry. The main challenge to achieve such objective is how to optimize performance of machine translation model targeted to limited resource of computing devices. Vanilla transformer model has been well known as one of state-of-the-art neural machine translation model. However, the drawback of this model is its large number of parameter models which might not be suitable for low-resource computing devices. This paper presents study findings in efforts to optimize 2 encoder-decoder stack depth of vanilla transformer by exploring several activation function using fine-tuning approach. The pre-trained transformer model is fine-tuned using parallel corpus Bahasa Indonesia-Sundanese language to address machine translation task. The experiment results found that Sigmoid gives the highest model performance (0.993 average training accuracy and 0.987 average testing similarity) and GeLU gives the lowest model performance (0.987 average training accuracy and 0.980 average testing similarity) of the tested vanilla transformer models.

 

Keywords

Neural Machine Translation, transformer model, low-resource computing.

Published in: 3rd Asia Pacific International Conference on Industrial Engineering and Operations Management, Johor Bahru, Malaysia

Publisher: IEOM Society International
Date of Conference: September 13-15, 2022

ISBN: 978-1-7923-9162-0
ISSN/E-ISSN: 2169-8767