Baseline Probabilities for Two-Alternative Forced Choice Tasks When Judging Stimuli in Evolutionary Psychology: A Methodological Note

Many paradigms in evolutionary psychology involve forced choice tasks with two alternatives. While the number of trials used across studies varies substantially, in such tasks it is common to test against a baseline of 50% (often via a one-sample t-test). In this paper, we simulate forced choice designs, varying in sample sizes (30 to 120) and number of trials (2 to 34) to empirically examine the usefulness of a 50% benchmark. Our results show that 50% is a weak benchmark when using a small number of trials. The simulations also indicate that increasing the number of trials is beneficial if one wants to use a 50% benchmark. There are however, marginal returns to increasing the number of trials: moving from 2 to 8 trials matters substantially more than moving from 28 to 34. Our approach also illustrates the value of simulations for understanding experimental designs, such as forced choice tasks, in evolutionary psychology.