Physical exertion monitoring has been strongly emphasized to avert the ill-effects of physically demanding nature of many industries such as construction. Recently, several sensors-based approaches have been suggested as an alternative to traditional subjective feedback-based methods. Although the proposed sensor-based approaches have laid the foundation for automated physical exertion monitoring, they require multiple on-body and/or off-body sensors to collect psychological, physiological, acceleration/posture or weather-related data. As such, multiple on-body sensors may instigate irritation and discomfort whereas other off-body sensors require additional resources for handling and managing them. To address these limitations, taking a minimalistic approach, this study explored the use of heart rate variability (HRV) metrics which could be computed from a single electrocardiogram or optical sensor (often found in fitness wrist bands and smart watches). For this purpose, manual material handling experiments were conducted while state-of-the-art HRV features were used to perform physical exertion monitoring with ensemble classifiers and artificial neural network (ANN) based regression analysis. The results indicate that ensemble classifiers achieved accuracies from 64.2% to 81.2%, depending on the number of levels in which physical exertion data was divided, whereas ANN regression achieved the least root mean square error of 1.651. Given the wide availability of HRV sensors in fitness bands and wrist watches, this study highlights the usability and limitations of HRV based physical exertion monitoring which could help make informed decisions related to its adoption in physically demanding industries such as construction.