A hierarchical strategy-optimized double deep Q-network for dynamic reentrant hybrid flow-shop scheduling problem with multi-stage batch processing machines

Due to the simultaneous consideration of operations sequencing and single/batch processing machine selection, the Reentrant Hybrid Flow-shop Scheduling Problem with Batch Processing Machine is a very complex NP-hard problem. In addition, the characteristics of large-scale, disturbance factors and multi-stage batch in actual production not only significantly increase complexity, but also require scheduling methods to have fast response capabilities. Based on the above characteristics and considering the new job arrival, this study develops a Dynamic Reentrant Hybrid Flow-shop Scheduling Problem with Multi-stage Batch Processing Machines (DRHFSP-MB) and proposes a Hierarchical Strategy-optimized Double Deep Q-network (HSDDQN) for solving it. First, by incorporating the self-attention mechanism, a hierarchical double deep Q-network structure is introduced to construct two agents, namely the batching agent and the scheduling agent, for solving the job batching and scheduling subproblems of DRHFSP-MB. In addition, to address the characteristics of multi-stage batch-processing and reentrant scheduling in DRHFSP-MB, a Markov decision process considering these two agents is designed, consisting of states, actions and rewards. Furthermore, a mask-based action selection strategy combined with the ε-greedy and a soft-start target network update strategy are developed to enhance the efficiency and generalization of HSDDQN. By comparing with existing rules and deep reinforcement learning methods, extensive experiments have shown the effectiveness of HSDDQN and the proposed improvement strategies in solving DRHFSP-MB.