Optimal transport filtering for robust cross-modal retrieval with open-set noisy labels

With the rapid growth of the Internet, cross-modal retrieval (CMR) has attracted increasing attention for its ability to bridge heterogeneous modalities. Most existing studies assume that data annotations are completely accurate, but this rarely holds in practice since both human non-expert and machine annotations often introduce noisy labels. To mitigate the effect of label noise, various robust learning approaches have been proposed with promising results. However, these methods are generally developed under a closed-set assumption, where noisy samples remain within the known label space. In real-world scenarios, noisy samples may originate from unseen categories, referred to as open-set noisy labels, under which existing methods often fail. To address this challenge, we propose a novel CMR framework named Optimal Transport Filtering with Open-Set Noisy Labels (OTOS). Specifically, a Discriminative Reinforcement Learning (DRL) module is introduced to enhance instance-level discrimination and reduce multimodal heterogeneity, while an Optimal Transport Filtering (OTF) module leverages geometric distances to identify clean samples, closed-set noise, and open-set noise effectively. Furthermore, specialized learning strategies are designed for different instance types to fully exploit the information contained in both closed-set and open-set data. Extensive experiments on three benchmark datasets demonstrate the superior performance of OTOS, particularly in handling multimodal data with open-set noisy labels.