The Gleason grading system assists in evaluating the prognosis of men with prostate cancer. Cancers with a higher score are more aggressive and have a worse prognosis. The pathologists observe the tissue components (e.g. lumen, nuclei) of the histopathological image to grade it. The differentiation between Grade 3 and Grade 4 is the most challenging, and receives the most consideration from scholars. However, since the grading is subjective and time-consuming, a reliable computer-aided prostate cancer diagnosing techniques are in high demand. This study proposed an ensemble computer-added system (CAD) consisting of two single classifiers: a) a specialist, trained specifically for texture features of the lumen and the other for nuclei tissue component; b) a fusion method to aggregate the decision of the single classifiers. Experimental results show promising results that the proposed ensemble system (area under the ROC curve (Az) of 88.9% for Grade 3 versus Grad 4 classification task) impressively outperforms the single classifier of nuclei (Az=87.7) and lumen (Az=86.6).