Exploring Virtual Environments Through Video Data: The RIVER Corpus and Beyond

The integration of AI-labelled technologies, such as Voice User Interfaces (VUI), Virtual Agents (VA), social robots, chatbots, AI clones, and other non-human actors, into organizational settings is growing rapidly. While some people see advancements like AI clones attending meetings on one’s behalf as breakthroughs, others fear AI misuse in fraud schemes. Broader discussions about AI technologies have sparked debates about the risks and ethical implications of using AI-generated entities in professional contexts. These debates raise critical questions about the visual literacy skills needed to detect deceptions and emphasize the importance of visual research methods required to study human – non-human interactions in virtual environments. Our presentation draws on insights from the RIVER (Real-Time Interactions in Virtual EnviRonments) corpus, a collection of video recordings capturing technology-mediated interactions within virtual teams in UK organisations. This resource enables systematic analysis of real-time online meetings in situ. We will demonstrate the application of multimodal conversation analysis (CA), a research method that examines talk, gestures, gaze, and other embodied actions to uncover processes of meaning-making. By illustrating how video data and transcriptions can serve as training tools, enhancing employees’ visual literacy, communication, and team management skills, we propose extending multimodal CA to study communication dynamics between humans and AI entities in hybrid environments. By fostering interdisciplinary dialogue, we aim to inspire further development of video datasets involving human and non-human participants. These initiatives could provide valuable insights into the opportunities and challenges posed by AI technologies in today’s rapidly evolving digital landscapes.