We propose and investigate for the first time a seamless millimeter-wave (mmW) radio-over-fiber (RoF) and radio-over-free-space optics (FSO)-based downlink for use in a passive optical network architecture using 4-, 16- and 64-quadrature amplitude modulation (QAM) for broadband wireless access (BWA) networks. The proposed system is implemented in both experiment and simulation to realize continuous and ubiquitous coverage in urban and rural areas. We outline, a proof-of-concept demonstration of 4-, 16- and 64-QAMs at data rates of 34, 67 and 100 Mb/s, respectively transmitted over a 15 km standard single-mode fiber (SMF), which is then optically up-converted to 25 GHz for transmission over a 10 km of SMF and a 2 m of FSO channel under a non-uniform turbulent condition. We show the measured error vector magnitude (EVM) values of 13, 9.2 and 7.3% for 4-, 16- and 64-QAM, respectively, which are below the corresponding standard EVM requirements and therefore confirm the practicality of the proposed hybrid system. Depending on the data rates, each modulation can be adaptively configured. We report a simulation of a 10 Gb/s 4- and 64-QAM hybrid RoF-FSO downlink under an extended non-uniform turbulence regime to verify the feasibility of the proposed scheme for use in practical applications. By implementing the decision-directed carrier phase recovery and linear electrical equalization, EVMs can be efficiently reduced below the required limits. We further evaluate the proposed system performance in terms of the bit error rates, constellation diagrams, received optical and mmW powers. Using state-of-the-art K-band power amplifiers and conical horn antennas, the maximum wireless transmission distance is estimated to be about 135 m for use in the last-mile BWA networks.