The appeal of video gaming has undoubtedly withstood the test of time. In view of its increasing popularity, lay people and researchers alike have taken an interest in the psychological consequences of video gaming. However, there seems to be a paradox associated with the effect of video gaming on gamers' well-being—namely, while most video game players cite “fun” as their motivation to play video games, video games continue to hold a notorious reputation among some researchers for being detrimental to mental health and emotional well-being as measured by indicators such as happiness, perceived stress, anxiety, and depressive symptoms. We suggest that a significant contributor to the mixed literature is the oversight of contextual factors that may moderate this relationship. The current review highlights five important contextual factors that should be considered when studying the associations between the frequency of video gaming and well-being. Specifically, we suggest that unless the social context (who), type (what), motivation (why), time and day (when), and amount (how much) of video gaming activities are adequately considered, examinations of well-being outcomes in relation to video gaming will remain incomplete.