Co-design with communities interested in heritage has oriented itself towards designing for polyvocality to diversify the accepted knowledges, values and stories associated with heritage places. However, engagement with heritage theory has only recently been addressed in HCI design, resulting in some previous work reinforcing the same realities that designers set out to challenge. There is need for an approach that supports designers in heritage settings in working critically with polyvocality to capture values, knowledges, and authorised narratives and reflect on how these are negotiated and presented in the designs created. We contribute “Designing with Genius Loci” (DwGL)—our proposed approach to co-design for polyvocality. We conceptualised DwGL through long-term engagement with volunteers and staff at a UK heritage site. First, we used ongoing recruitment to incentivise participation. We held a series of making workshops to explore participants’ attitudes towards authorised narratives. We built participants’ commitments to collaboration by introducing the common goal of creating an interactive digital design. Finally, as we designed, we enacted our own commitments to the heritage research and to participants’ experiences. These four steps form the backbone of our proposed approach and serve as points of reflexivity. We applied DwGL to co-creating three designs: Un/Authorised View, SDH Palimpsest and Loci Stories, which we present in an annotated portfolio. Grounded in research through design, we reflect on working with the proposed approach and provide three lessons learned, guiding further research efforts in this design space: (1) creating a conversation between authorised and personal heritage stories; (2) designing using polyvocality negotiates voices; and (3) designs engender existing qualities and values. The proposed approach places polyvocality foremost in interactive heritage interpretation and facilitates valuable discussions between the designers and communities involved.