Due to competition a good number of E-commerce sites are trying to improve their systematic infrastructure according to customers' choices. Coping up with this, E-commerce companies are facing challenges in the business environment of Bangladesh. Our study is to analyze the challenges of our current e-commerce sector in Bangladesh from both customer and company point of view. From the secondary data, seven challenging factors (customers’ point of view) and seven challenging factors (company’s point of view) were selected. Firstly, two different surveys were launched and Likert scale was analyzed.  By one sample t-test, some challenges were rejected due to their low significance value. The RI (Relative importance index) method was applied for the rest of the factors. According to the RII value, challenging factors were ranked. In both perspectives, Trust factor has been ranked as the most significant challenge. Other factors such as product quality, price issue etc. (customers’ perspective) and strategic issues, transportation issues etc. (Company’s perspective) have been ranked as very important challenges. Necessary steps should be taken to overcome these challenges according to their importance.