An advanced nonlinear continuum model is presented to analyse the super and subcritical nonlinear behaviour of nanotubes. The nanoscale system is used to convey fluid flow at nanoscale levels. Due to the restrictions of one-parameter size-dependent models, a more comprehensive nonlinear coupled model containing two different size parameters is introduced using the nonlocal strain gradient theory (NSGT). Both axial and transverse inertial terms are taken into consideration, leading to more accurate results for nanotubes conveying fluid. In addition, since the mean free path of molecules is not negligible compared to the diameter of the tube at nanoscales, the Beskok–Karniadakis approach is implemented. The NSGT, Galerkin’s technique and continuation method are finally employed to derive, discretise and solve the coupled nonlinear equations, respectively. The frequency–amplitude response, modal interactions and the possibility of energy transfer between modes are examined in both supercritical and subcritical flow regimes.