Autonomous navigation is of great importance for service robots. Such robots need various technologies, especially localization, and mapping. In this paper, we focus on the localization problem. Commonly, to solve the localization problem, the creation of a map is needed. However, creating the map takes considerable time and costs. Therefore, one solution for indoor environment is to use architectural floor plans since buildings have own floor plans. If a robot can use them for localization, it enables users to cut the time and costs for creating the map from scratch. However, the floor plans sometimes do not match with a real building. Besides, sensor measurement sometimes contains objects such as a pedestrian, which are not contained in the floor plans. In this paper, we propose a localization algorithm with architectural floor plans that is robust to such inconsistencies. The algorithm estimates a robot coordinate by matching the floor plan with the point clouds obtained from depth images. Outliers derived from the inconsistencies in the point clouds are filtered with plane information from the depth images with the Generalized ICP framework. We tested our algorithm with floor plans in a real building and in a simulator as a case study. The results show that our algorithm can localize a robot with average more than twice accuracy compared to AMCL and be used for real-time applications.