The presence of exudates is one of the most significant signs of Diabetic retinopathy (DR) whereas; white or tiny yellow deposits known as drusen mostly identify age-related macular degeneration (AMD). Exudates and drusen may share a similar appearance; hence discriminating them is of extreme importance in enhancing automated AMD and DR diagnosis. Fortunately, diagnosing these diseases in their early stages is extremely useful for effective treatment since they are usually treatable. The goal of this research is to develop an automated tool that helps the pathologist diagnose the type of disease correctly and distinguish between DR, AMD, and normal fundus images through accurate classification of exudates and drusen lesions. In this paper, an automatic retinal diagnosis system that combines different texture and colour features is proposed. New textural and colour features are used in a bag-of-features approach for efficient and accurate detection. A codebook is generated using a bagged combination of inter colour local binary pattern (ICLBP) and colour vector angles (CVA) features to exploit textural and colour information for efficient and accurate classification. Intensive experiments show that the proposed dictionary learning-based system can capture the variety of structures and patterns in retinal fundus images and produce discriminant descriptors for classification. Using an SVM classifier with the obtained bagged combination of the proposed ICLBP and CVA features, the system has been shown to offer high classification performance. The experimental performance has been obtained with a dataset of 798 retinal images collected from various standard datasets, namely: DIARETDB0, DIARETDB1, HEI-MED, STARE, and MESSIDOR. All experiments were conducted with 10-fold cross validation using the classification accuracy, sensitivity, specificity, and area under curve. Correct classification is reported with an average sensitivity of 98.37%, specificity of 99.64% and accuracy of 99.67% and an overall average area under the curve of 0.983%. This represents the best performance achieved so far when compared to existing state-of-the-art systems for the diagnosis of retinal disease with drusen and exudates being the key characteristics in fundus image classification.