The problems that arise in multitudinous fields often involve solving complex nonlinear ordinary differential equations (ODE), and it remains challenging since the actual solutions to these problems are hard to obtain. In this regard, the solution strategy with the formulation of Fourier series expansion, calculus of variation and metaheuristic algorithm, was introduced to determine the approximate solution of the nonlinear ODE. The nonlinear ODE was formulated as an optimization problem, specifically, the mothflame optimization (MFO) algorithm and flower pollination algorithm (FPA) were utilized to find the coefficients of the Fourier series. This paper aimed to determine the feasibility of the proposed method to solve the ODEs with different characteristics and compare the obtained results with other optimization algorithms. Moreover, the suitable number of terms (NT) of Fourier series were determined for different test problems for MFO and FPA. The quantitative analysis in terms of the generational distance (GD) metric demonstrated that the approximate solutions were reasonably accurate, with the low GD within the range of 1E-03 to 1E-05 for all test problems. The comparative analysis showed that the approximate performances of MFO and FPA were superior to or comparable with the genetic algorithm, particle swarm optimization and water cycle algorithm.