Social isolation due to the coronavirus disease 2019 (COVID-19) pandemic has reduced physical activity levels in both men and women. The identification of barriers to physical activity may assist in developing strategies to increase levels of physical activity during this pandemic. The study aim was identify the barriers to regular participation in physical during the COVID-19 pandemic in Brazilian adults. This cross-sectional study included 1570 [56.6% women; aged: 39.1 (37.7–40.7) years old] in social isolation due COVID-19. Barriers to physical activity were obtained using the validated questionnaires. “Laziness and fatigue” (50.2%), “lack of motivation” (31.2%), “lack of appropriate facilities/equipment/space” (17.4%), and “lack of time” (13.0%) were the barriers most prevalent in the study. Lack of motivation (OR = 1.49; 95% CI = 1.19–1.86) and lack of appropriate facilities/equipment/space (OR = 2.11; 95% CI = 1.57–2.83) were most associated with impacting physical activity levels due to the COVID-19, independent of sex, age, education level, days of social isolation and status weight. In conclusion, personal barriers to physical activity are common between both sexes, with lack of motivation and lack of appropriate facilities/equipment/space most associated with a decreased level of physical activity due to the COVID-19 pandemic.