4D image construction of thoracic dynamic MRI data provides clinicians the capability of examining the dynamic function of the left and right lungs, left and right diaphragms, and left and right chest wall separately. For the method implemented based on free-breathing rapid 2D slice acquisitions, often part of the acquired data cannot be used for the 4D image reconstruction algorithm because some patients hold their breath or breathe in patterns that differ from regular tidal breathing. Manually eliminating abnormal image slices representing such abnormal breathing is very labor intensive considering that typical acquisitions contain ∼3000 slices. This paper presents a novel respiratory signal classification algorithm based on optical flow techniques and a SVM classifier. The optical flow technique is used to track the speed of the diaphragm, and the motion features are extracted to train the SVM classification model. Due to the limited number of abnormal samples usually observed, 118 abnormal signals were generated by simulation by appropriately transforming the normal signals, so that the number of normal and abnormal signals reached 160 and 160, respectively. In the process of model training, our goal is to reduce the error rate of false negative abnormal signal detection (FN) as much as possible even at the cost of increasing false positive misclassification rate (FP) for normal signals. From 10 experiments we conducted, the average FN rate and FP rate reached 5% and 26%, respectively. The accuracy over all (real and simulated) samples was 85%. In all real samples, 82% of the abnormal data were correctly detected.