The transformation of systematic vacuum and hydrogen annealing effects in graphene devices on the SiO2 surface is reported based on experimental and van der Waals interaction corrected density functional theory (DFT) simulation results. Vacuum annealing removes p-type dopants and reduces charged impurity scattering in graphene. Moreover, it induces n-type doping into graphene, leading to the improvement of the electron mobility and conductivity in the electron transport regime, which are reversed by exposing to atmospheric environment. On the other hand, annealing in hydrogen/argon gas results in smaller n-type doping along with a decrease in the overall conductivity and carrier mobility. This degradation of the conductivity is irreversible even the graphene devices are exposed to ambience. This was clarified by DFT simulations: initially, silicon dangling bonds were partially terminated by hydrogen, subsequently, the remaining dangling bonds became active and the distance between the graphene and SiO2 surface decreased. Moreover, both annealing methods affect the graphene channel including the vicinity of the metal contacts, which plays an important role in asymmetric carrier transport.