Load balancing is critical for achieving high performance and availability in distributed object storage systems. Traditional static algorithms such as round-robin and least-connections provide predictable behavior but cannot adapt to dynamic workload conditions. While machine learning-based approaches promise adaptive routing, they often introduce prohibitive computational overhead. This paper presents a lightweight dynamic load balancing system integrating XGBoost gradient boosting with HAProxy, an open-source load balancer and MinIO distributed object storage, evaluated through the lens of four testable hypotheses. We hypothesize and empirically verify that: (H1) ML (machine learning) latency optimization produces a measurable throughput tradeoff due to load concentration; (H2) routing effectiveness varies significantly by S3 (Simple Storage Service) operation class, with metadata and write operations benefiting more than reads; (H3) a hybrid strategy selectively applying ML to writes and metadata while using static algorithms for reads outperforms any single uniform strategy; and (H4) a cached XGBoost inference pipeline can deliver routing decisions within 2ms, making ML-based load balancing computationally feasible for production use. Through extensive evaluation on a four-node MinIO cluster — benchmarking the ML system against nine static algorithms including round-robin and least-connections — we confirm H1 (DELETE achieves best-in-class latency at 11.12ms but -20% throughput), H2 (GET operations show a 52% latency gap while STAT achieves best-in-class 4.98ms), and H4 (0.5-2ms inference overhead at 10-30% CPU utilization). H3 is supported analytically and proposed as a primary direction for future implementation. This work provides an open-source foundation for adaptive load balancing research in distributed storage environments.
Lightweight Machine Learning-Based Dynamic Load Balancing for Distributed Object Storage Systems
21 views
2 Downloads