In pandemics like Covid-19, the use of autonomy and machine learning technologies are of high importance. Internet of things (IoT) enabled autonomous transportation system (ATS) envisions a fundamental change in the traditional transportation system. It aims to provide intelligent and automated transport of passengers, goods, and services with minimal human interference. While ATS targets broad spectrum of transportation (Cars, trains, planes etc.), the focus of this paper will be limited to the use of vehicles and road infrastructure to support healthcare and related services. In this paper, we offer an IoT based ATS framework for emergency healthcare services using Autonomous Vehicles (AVs) and deep reinforcement learning (DRL). The DRL enables the framework to identify emergency situation smartly and helps AVs to take faster decision in providing emergency health aid and transportation services to patients. Using ATS and DRL for healthcare mobility services will also contribute towards minimizing energy consumption and environmental pollution. This paper also discusses current challenges and future works in using ATS for healthcare services.