​Active Defect Detection With Human-in-the-loop Deep Learning

Abstract

Accurate defect detection is critical for ensuring the quality and operational efficiency of steel production. Traditional methods, such as manual inspection, are both labor-intensive and prone to human error. Moreover, the scarcity of annotated datasets poses a significant challenge for developing robust machine learning models in this domain. This research introduces a novel, data-efficient framework that leverages cutting-edge deep learning techniques to address these limitations, significantly enhancing defect detection processes in industrial settings.
The core contribution of this work is a Deep Bayesian Active Learning framework for steel defect classification, integrating Monte Carlo (MC) Dropout with pseudo-labelling and a humanin- the-loop system. The methodology alternates between active learning and pseudo-labelling, significantly reducing the labelling requirements. By selecting the most uncertain data points for annotation, we minimise the need for large annotated datasets while maintaining model accuracy. The pseudo-labelling process is iteratively alternated with active learning, allowing the model to continually improve from both labelled and pseudo-labelled data. Our experiments show that the framework achieves high classification accuracy with only 20% of the dataset labelled, while tolerating minor errors in the pseudo-labels. This innovative integration of human-in-the-loop annotation and deep learning greatly reduces the cost and time involved in defect detection.
In a subsequent chapter, we extend the use of Bayesian Active Learning to the domain of object detection by incorporating the Single Shot Multibox Detector (SSD) architecture. We adapt SSD with active learning to effectively localise defects within steel images, significantly improving detection efficiency. Our results demonstrate that using only 10% of the labelled data, we achieve a mean average precision (mAP) of approximately 42%, underscoring the effectiveness of this approach in real-world scenarios. This chapter shows that by combining active learning with SSD, defect localisation can be made more efficient without sacrificing accuracy, making it highly applicable for industrial automation tasks.
Additionally, this research introduces MDC-Net (Multimodal Detection and Captioning Network), which tackles defect detection as a multimodal problem by integrating object detection with natural language processing. Using a Transformer-based architecture, we apply an encoder-decoder model in an autoregressive manner to generate descriptive captions for detected defects alongside their bounding box coordinates. The decoder predicts the next token in the sequence based on previously predicted tokens, effectively treating defect captioning as a natural language generation task. This novel combination of object detection and captioning enhances human decision-making by providing contextual descriptions of the detected defects, automating not just the detection but also the interpretation process. Notably, this approach eliminates the need for complex post-processing techniques, further simplifying the overall workflow.
The empirical validation of our methodologies was conducted on the NEU steel defect dataset, where we observed substantial improvements in detection accuracy and operational efficiency compared to traditional approaches. By automating both defect detection and the generation of descriptive captions, our framework streamlines inspection workflows by reducing the burden on human inspectors, ultimately leading to enhanced operational performance in industrial settings.
In conclusion, this thesis contributes to the field of defect detection by presenting a suite of advanced deep learning methods that drastically reduce labelling requirements, improve detection accuracy, and providing an efficient, scalable solution for industrial environments. The integration of novel methodologies, including human-in-the-loop, multimodal transformers, and pseudo-labelling, sets a strong foundation for the next generation of defect detection systems in manufacturing.

Date of Award	28 Nov 2024
Original language	English
Awarding Institution	Northumbria University
Supervisor	Wai Lok Woo (Supervisor), Edmond Ho (Supervisor) & Shanfeng Hu (Supervisor)