A number of vision-based algorithms designed to detect and track unmanned aerial vehicles (UAVs) from on board a second UAV have been researched, implemented, and experimentally validated over the last decade. However, the successful methods have tended to rely on characteristics such as color or shape, meaning they require the target UAV to have particular markings or geometries. This paper uses the Viola–Jones cascade classifier, a computer vision algorithm originally designed to detect human faces in video streams, and demonstrates its capability for detecting and tracking an arbitrary type of UAV with excellent performance in either indoor or outdoor environments and with a variety of backgrounds. The Viola–Jones algorithm is applied to two specific quadrotor UAV models, the Solo from 3D Robotics and the AR.Drone 2.0 from Parrot. Experimental testing demonstrates that the resulting system achieves very good detection and tracking performance in real time on each UAV type for both indoor and outdoor flight tests.