The maintenance of the undifferentiated state in human embryonic stem cells (hESCs) is critical for further application in regenerative medicine, drug testing and studies of fundamental biology. Currently, the selection of the best quality cells and colonies for propagation is typically performed by eye, in terms of the displayed morphological features, such as prominent/abundant nucleoli and a colony with a tightly packed appearance and a well-defined edge. Using image analysis and computational tools, we precisely quantify these properties using phase-contrast images of hESC colonies of different sizes (0.1–1.1 mm2) during days 2, 3 and 4 after plating. Our analyses reveal noticeable differences in their structure influenced directly by the colony area A. Large colonies (A > 0.6 mm2) have cells with smaller nuclei and a short intercellular distance when compared with small colonies (A < 0.2 mm2). The gaps between the cells, which are present in small and medium sized colonies with A ≤ 0.6 mm2, disappear in large colonies (A > 0.6 mm2) due to the proliferation of the cells in the bulk. This increases the colony density and the number of nearest neighbours. We also detect the self-organisation of cells in the colonies where newly divided (smallest) cells cluster together in patches, separated from larger cells at the final stages of the cell cycle. This might influence directly cell-to-cell interactions and the community effects within the colonies since the segregation induced by size differences allows the interchange of neighbours as the cells proliferate and the colony grows. Our findings are relevant to efforts to determine the quality of hESC colonies and establish colony characteristics database.