Abstract: The selection of effective features is always the key issue of speech emotion recognition algorithm. For the problem of the speech emotion features selection and construction, one speech emotion recognition algorithm based on the selective attention mechanism is proposed. Firstly, considering the time-frequency of speech, the spectrogram is calculated. Secondly, imitating the selective attention mechanism, the color, direction and brightness map of the spectrogram is computed. Based on these maps, the feature matrix is formed after normalization. Then, the feature matrix is rearranged and reduced the dimensionality by the principal component analysis (PCA) to form the emotion recognition feature vector. Finally, the speech emotion is classified with the improved support vector machine (SVM). Experiments on emotion (anger, fear, joy, sadness and surprise) recognition show the proposed method has an improved recognition rate 85.44%. Compared with the prosody and voice quality features, the speech emotion recognition rate has been improved at least by 10%. And compared with others spectrogram features, the improved recognition rate reaches 7%.