Abstract: The speaker clustering is a key component in many speech applications. An improved speaker clustering algorithm based on hierarchical clustering was presented. Compared with typical hierarchical clustering approaches, the proposed method improves the performance of the clustering from three aspects. First, a modified sum-of-squared-error criterion is implemented to ensure better efficiency of the clustering. Then hypothesis test is introduced to select the best clusters automatically. Additionally, a robust online speaker clustering algorithm based on hypothesis test is described. The clustering decision can be made whenever a new audio segment is received. Experiments indicate that the improved hierarchical speaker clustering achieves good performance on both precision and speed. The method obtains average speaker purity of 96.7% and average cluster purity of 96.6%. Experiments also show that this method is effective at improving the performance of the unsupervised adaptation even comparing with the true speaker-condition.