Thermography is an established technique in Non-Destructive Testing and Evaluation (NDT&E) for detecting and visualizing defects. The acquired thermal image sequences contain valuable information in the spatial, time and frequency domains. Many thermal feature extraction methods have been previously proposed to detect and quantify defects. However, the quantitative assessment and error analysis remain as challenges. In the present work, a quantitative analysis strategy is proposed for validating detection performance of various thermal feature extraction techniques based on Eddy Current Stimulated Thermography. F-score is adopted as a global quantitative evaluation indicator to assess the crack detectability. The quantitative validation is carried out based on a large number of simulations and experiments for cracks on the nonferromagnetic material. The detectability in relation to the variations in crack size and the different thermal feature extraction methods have been critically analyzed. The insights obtained from the present findings can serve as guidelines to other types of thermal-based NDT&E to improve the defect visibility and detectability.