Microtubules in mammalian cells are cylindrical protein polymers which structurally and dynamically organize functional activities in living cells. They are important for maintaining cell structures, providing platforms for intracellular transport, and forming the spindle during mitosis, as well as other cellular processes. Various in vitro studies have shown that microtubules react to applied mechanical loading and physical environment. To investigate the mechanisms underlying such phenomena, a mathematical model based on the orthotropic higher-order shear deformation shell formulation and Hamilton's principle is presented in this paper for dynamic behavior of microtubules. The numerical results obtained by the proposed shell model are verified by the experimental data from the literature, showing great consistency. The nonlocal elasticity theory is also utilized to describe the nano-scale effects of the microtubule structure. The wave propagation and vibration characteristics of the microtubule are examined in the presence and absence of the cytosol employing proposed formulations. The effects of different system parameters such as length, small scale parameter, and cytosol viscosity on vibrational behavior of a microtubule are elucidated. The definitions of critical length and critical viscosity are introduced and the results obtained using the higher order shell model are compared with those obtained employing a first-order shear deformation theory. This comparison shows that the small scale effects become important for higher values of the wave vector and the proposed model gives more accurate results for both small and large values of wave vectors. Moreover, it is shown that for higher circumferential wave number, the torsional wave velocity obtained by the higher-order shell model tend to be higher than the one predicted by the first-order shell model.