Abstract: In order to improve the quality of automatic fiber placement and assist on-site personnel to quickly detect defects, this paper proposes a real-time instance segmentation network named Trans-Yolact, which is based on Transformer. The Trans-Yolact is used to detect, classify and segment multi-spectrum images of composite material defects. Based on Yolact, aiming at the characteristics of composite material defects, Trans-Yolact's detection ability of composite material defects is enhanced from the two dimensions of space domain and channel domain. In the spatial domain, the convolution kernels have the limitation of spatial scale. The detection of narrow, long, large-size defects is not effective. Therefore, this paper adopts the BoTNet of the CNN+Transformer architecture as backbone; at the same time, the Transformer is introduced into the FPN structure of the Yolact network to enhance the network's ability to obtain information from non-local spaces. In the channel domain, the infrared and visible simultaneous detection method is adopted, and the shallow structure of the backbone is improved, which is divided into visible channel, infrared channel, and mixed channel. Channel domain attention mechanism is introduced in mixed channel. Enhance the comprehensive judgment ability of the network for infrared and visible images. The results show that the mAP of Trans-Yolact for composite defect detection is 88.0%, which is 5.5% higher than Yolact network, and the AP of narrow defects such as miss and twist are increased by 15.2% and 5.1%. The AP of foreign defects including some large-scale defects is increased by 9.1%. Finally, the Trans-Yolact network is pruned. After pruning, the amount of floating-point operations per second (FLOPs) and parameters are reduced by 26.5% and 44.7% compared with Yolact network. The number of detection frames is increased by 58%, reaching 57.67 fps. And the online test is carried out on the large-scale gantry composite material automatic laying equipment, which can meet the real-time detection and segmentation of composite material defects under the maximum laying speed of 1.2 m/s in the production process.