Abstract: A visual tracking method via object detection based on deep learning was proposed. In consideration of the advantages of deep learning in feature representation, deep model SSD(Single Shot Multibox Detector) was used as the candidate object extractor in the tracking model. Simultaneously, the color histogram feature and HOG(Histogram of Oriented Gradient) feature were combined to select the tracking object. In the process of tracking, multi-scale object searching map, which was applied to implement the object detection in different scales, was built to improve the detection performance of deep learning model. In the experiment of eight respective tracking video sequences in the baseline dataset, compared with six typical tracking methods, the proposed method has better performance in tracking effect, and has better robustness in the tracking challenging factors, such as deformation, scale variation, rotation variation, illumination variation, and background clutters.