Abstract: A method of voice conversion based on self organization clustering and determining parameters of conversion model by modified Particle Swarm Optimization (PSO) is proposed. Firstly, Self Organization Feature Mapping (SOFM) Network is used to cluster the characteristic parameters, and then the conversion rule for each cluster is established, where the parameters of conversion model in each conversion rule are determined by modified PSO with Cauchy mutation. Compared with the single conversion rule in conventional method, the multiple rules established by using clustering and parameters determined by modified PSO can improve the accuracy of mapping function and avoid the model parameters being trapped in the local optimum. The experiments take the conversion from female to male as example, by subjective evaluation, the proposed method increases the similarity by 27.6% through ABX test, and increases the Mean Opinion Score (MOS) by 0.6, and by objective evaluation, the spectral distortion with proposed method is the least.