With the rise in e-commerce and growing consumer demand, supply chains are striving to achieve cost-efficient delivery operations. One of the most challenging distribution problems is the last-mile delivery which generally refers to the very last leg of the delivery operation when a parcel is moved from a depot to the end customer. Despite the significant increase in the number of products ordered online and delivered directly to customers, the reliance on truck-based delivery has remained unchanged for decades. The integration of drones in the last-mile delivery can overcome many operational challenges and provide competitive advantages. Traditional optimization algorithms for truck-drone suffer from high computational time and can solve only small-sized problems. This study focuses on finding an effective approach for solving the last-mile delivery problem. We propose a hybrid machine learning model to tackle the problem of delivering products to a set of delivery locations using a modern truck equipped with multiple drones. A constrained clustering algorithm is proposed to cluster delivery locations based on user-specified constraints in the first stage. The center of each constrained cluster serves as a launching site where the truck can park, and drones can be dispatched to make all final deliveries. A deep reinforcement learning model is proposed in the second stage to find an optimal route among all clusters. Extensive experiments are conducted to evaluate the proposed approach. Solutions obtained are compared with other traditional algorithms. Results show that our approach can produce quality solutions in real-time.