A multi-objective optimization of in situ sol-gel process was conducted in preparing oil palm fiber-reinforced polypropylene (OPF-PP) composite for an enhancement of mechanical and thermal properties. Tetraethyl orthosilicate (TEOS) and butylamine were used as precursors and catalysts for the sol-gel process. The face-centered central composite design (FCCD) experiments coupled with response surface methodology (RSM) has been utilized to optimize in situ silica sol-gel process. The optimization process showed that the drying time after the in-situ silica sol-gel process was the most influential factor on silica content, while the molar ratio of TEOS to water gave the most significant effect on silica residue. The maximum silica content of 34.1% and the silica residue of 35.9% were achieved under optimum conditions of 21.3 h soaking time, 50 min drying time, pH value of 9.26, and 1:4 molar ratio of TEOS to water. The untreated oil palm fiber (OPF) and silica sol-gel modified OPF (SiO2-OPF) were used as the reinforcing fibers, with PP as a matrix and maleic anhydride grafted polypropylene (MAgPP) as a compatibilizer for the fiber-reinforced PP matrix (SiO2-OPF-PP-MAgPP) composites preparation. The mechanical and thermal properties of OPF-PP, SiO2-OPF-PP, SiO2-OPF-PP-MAgPP composites, and pure PP were determined. It was found that the OPF-S-PP-MAgPP composite had the highest toughness and stiffness with values of tensile strength, Young's modulus, and elongation at break of 30.9 MPa, 881.8 MPa, and 15.1%, respectively. The thermal properties analyses revealed that the OPF-S-PP-MAgPP exhibited the highest thermally stable inflection point at 477 °C as compared to pure PP and other composites formulations. The finding of the present study showed that the SiO2-OPF had the potential to use as a reinforcing agent to enhance the thermal-mechanical properties of the composites.