Neighborhood environment and indoor environment are two main living environments to older adults during their daily life. Until now, many researchers have already paid attention to the improvement in the indoor environment for older adults, but few researchers notice that neighborhood environment also plays a critical role in the quality of life (QoL) of older adults, especially older adults who are aging in place. This article aims to explore the complex inner relationship between neighborhood environment and QoL of older adults based on systematic literature review (SLR) and interpretative structural modeling (ISM). All neighborhood environmental factors related to QoL of older adults are searched, screened, quality assessed, counted, and then identified according to SLR protocol. Then, based on the questionnaire survey and the one-to-one interview, the ISM of all identified neighborhood environmental factors is established to clarify the inner relationship among those factors in urban areas, and the role of each neighborhood environmental factor is analyzed by MICMAC analysis. This inner relationship among neighborhood environmental factors can provide valuable retrofit strategies of neighborhood environment to improve QoL of older adults efficiently.