Track: Systems Engineering

Abstract

The amount of data generated by traditional business activities, creating data repositories ranging from terabytes to petabytes in size. However, this information cannot be practically analyzed on a single commodity computer because the data is too large to fit in memory. For this purpose, the large size of data to be processed requires the use of high-performance analytical systems running on distributed environments. Because the data is so big it affects the types of algorithms we are willing to consider. Then standard analytics algorithms need to be adapted to take advantage of cloud computing models which provide scalability and flexibility.This paper introduces a new distributed training method, which combines the widely used framework for bigdata, MapReduce, with the traditional structure of multiple linear regression.Parallel processing of multiple linear regression will be based on the QR decomposition and the ordinary least squares method adapted to MapReduce. Our platform is deployed on Cloud Amazon EMR service.Experimental results demonstrate that the our parallel version of the multiple linear regression can efficiently handle very large datasets on commodity hardware with a good performance on different evaluation criterions, including number, size and structure of machines in the cluster.