Aiming at the bridge monitoring system, some of the monitoring data are abnormal due to equipment failure and environmental impacts. In this paper, the time-frequency domain convolutional neural network method is applied to the calculation of monitoring data and the risk assessment of bridge structure. The data collected by the acceleration sensor is firstly sliced and sampled and visualized. Then wavelet analysis is used to preprocess the cluttered data, and Wigner-Ville distribution and Fast Fourier Transform are introduced to extract time-frequency features from the collected data. A convolutional neural network is proposed and the network is trained on dual channel images fusing time and frequency domain images. By analyzing the spectrogram and and time-frequency diagram of the bridge monitoring data, the method of this paper classifies the bridge health condition into three kinds: no disease, slight disease and disease, which can accurately determine the health condition of different bridges, and the assessment accuracy of the risk assessment model based on the fusion of time-frequency domain information reaches 97.78%, which indicates that the high efficiency and feasibility of the bridge inspection data computation and the risk assessment model in this paper can meet the actual engineering construction needs of bridge inspection.