Abstract: In order to consider the effect of branch line integral for acoustic field in a range-dependent waveguide with varying topography, a coupled-mode sound propagation model with complex effective depth is presented. The equations of motion and continuity are used to obtain the coupled equations, which satisfy correct boundary conditions in the waveguide with varying topography and contain one coupling matrix. Meanwhile, the couplings between discrete and continuous spectrums are dealt with based on complex effective depth theory. It is shown with numerical simulations that the accuracy of transmission loss is improved by the coupled mode model when eigenvalue of trapped mode is located on the neighbourhood of branch point. The acoustic field in a non-horizontally stratified waveguide can be calculated efficiently and accurately by this model, and the energies corresponding trapped modes, leaky modes and branch line integral can be considered adequately.