Feature Integration Theory proposed that attention shifted between target-like representations in our visual field. However, the nature of the representations that determined what was target like received less specification than the nature of the attention shifts. In recent years, visual search research has focused on the nature of the memory representations that we use to guide our shifts of attention. Sensitive measures of memory quality indicate that the template representations are remembered better than other, merely maintained, memories. Here we tested the hypothesis that we prepare for difficult search tasks by storing a higher fidelity target representation in working memory than we do when preparing for an easy search task. To test this hypothesis, we explicitly tested participants’ memory of the target color they searched for (i.e., the attentional template) versus another memory that was not used to guide attention (i.e., an accessory representation) following blocks of searches with easy-to-find targets (i.e., distractors were homogeneously colored) to blocks of searches with hard-to-find targets (i.e., distractors were heterogeneously colored). Although homogeneous-distractor searches required minimal precision for distractor rejection, we found that templates were still remembered better than accessories, just like we found in a heterogeneous-distractor search. As a consequence, we suggest that stronger memories for templates likely reflects the need to decide whether new perceptual inputs match the template, and not an attempt to create a better template representation in anticipation of difficult searches.