Automated water-quality monitoring systems rely on human analysts to receive and act upon sensor alarms that can have important consequences for human and ecosystem health. Threshold-based alarms can lead to high false alarm rates that tax operators' cognitive load and decrease reliance (trust) on decision-support technology. Here we describe a human-in-the-loop anomaly detection framework for ground and surface-water applications that leverages contextual baseline models of normal behavior and interpretable machine learning to improve alarm accuracy. Time series sensor data were utilized to train flexible normal ranges, providing contextual ground truth for labeling anomalous sensor values. Predictive models including Logistic Regression (LR) and Random Forest (RF) were applied to determine interpretability-sensitivity tradeoffs. Our results indicate that RF increased the sensitivity of anomaly detection, whereas LR provided explainable probability scores that allow for human-understandable decisions. Using signal detection theory, we frame the utility of interpretability in automated decision support to decrease false alarms, alleviate cognitive load and increase decision confidence in human-in-the-loop environmental monitoring.