Social distancing rules during the COVID-19 pandemic changed social interaction for many and increased the risk of loneliness in the general population. Social media use has been ambiguously related to loneliness, and associations may differ by age. The study aimed to examine loneliness and its association with social media use within different age groups during the COVID-19 pandemic. A cross-sectional online survey was conducted in Norway, the United Kingdom, the United States, and Australia during April/May 2020, and 3,810 participants aged 18 years or above were recruited. Multiple regression analyses were conducted to examine associations between social media use and social and emotional loneliness within separate age groups. Emotional loneliness was higher among young adults and among those who used social media several times daily. Adjusting by sociodemographic variables, using more types of social media was associated with lower social loneliness among the oldest participants, and with higher emotional loneliness among the youngest participants. Among middle-aged participants, using social media more frequently was associated with lower social loneliness. We found that the associations between social media use and loneliness varied by age. Older people’s engagement on social media may be a resource to reduce loneliness during the COVID-19 pandemic. We observed higher levels of loneliness among high-frequent social media users of younger age.