Abstract: The scattering problem of a rough interface in a waveguide under small Rayleigh's parameter approximation is discussed in this paper. The derived scattering intensity can be written in the formI_R=\sum\limits_l, j I_l_j^s(l,j=1,2,3,…). Where I_lj^s is propo- rtional to m_lj^s which is defined as the normal mode scattering coefficient for the rough interface. A relation has been found between m_lj^s and the plane wave scattering coefficient m_s: m_l j^s=(2 \pi)^2 m_s /\left|V\left(\theta_l\right) V\left(\theta_j\right)\right| in which V(θ_l) and V(θ_j) represent the reflection coefficients respectively for incident angles θ_l and θ_j.