Abstract: The end-to-end acoustic modeling method based on connectionist temporal classification (CTC) criterion is studied and optimized in this paper. We study on the performance of CTC acoustic models with different acoustic features, modeling units and architectures. A modeling unit related unshared blank method is proposed to improve the modeling defects caused by the blank sharing in the CTC model. And a model initialization method that put the association information between the modeling units into the neural network is introduced to further improves the performance of the CTC model. Experiments were carried out on the 300-hour Switchboard dataset. Results show that the proposed CTC model trained with non-shared blanks, time-delay neural networks and the initialization method with association information between the modeling units achieves an absolute 1.1% reduction in word error rate as well as a 3.3-time speedup over the baseline system. The experimental results show that the proposed method is effective for end-to-end acoustic modeling.