Attentional control is thought to play a critical role in determining the amount of information that can be stored and retrieved from visual working memory (VWM). We tested whether and how task-irrelevant feature-based salience, known to affect the control of visual attention, affects VWM performance. Our results show that features of a task-irrelevant color singleton are more likely to be recalled from VWM than non-singleton items and that this increased memorability comes at a cost to the other items in the display. Furthermore, the singleton effect in VWM was negatively correlated with an individual’s baseline VWM capacity. Taken together, these results suggest that individual differences in VWM storage capacity may be partially attributable to the ability to ignore differences in task-irrelevant physical salience.