Multiple Scales Fusion and Query Matching Stabilization for Detection with Transformer

Recent advances in object detection with Transformer-based models like Detection with Transformer (DETR) have improved performance, but challenges remain with multi-scale fused features. These features introduce redundant tokens and bias toward larger objects, slowing down training. To overcome these issues, we propose two novel encoders: the Similarity-based Deduplication Encoder (SDE) and the Hybrid Multi-object Encoder (HMoE). HMoE employs an offset-based attention window to enhance local attention for objects of varying sizes across feature maps, while SDE reduces redundancy by calculating attention scores across multiple scales. Additionally, we introduce a One-to-many Positive Matching (OmPM) strategy to improve query stability. OmPM generates query vectors from multiple positive samples, resulting in more diverse and semantically meaningful queries. Our model demonstrates substantial performance improvements. On the Visual Object Classes Challenge 2007 dataset, it achieves a +5.04 mean Average Precision (mAP) and +5.1 Average Precision for small objects (APs) for small objects in just 24 epochs. On the Microsoft Common Objects in Context (COCO) dataset, the model reaches 50.1 mAP and 34.2 APs in only 8 epochs, and 52.4 mAP and 35.6 APs in 24 epochs. This significantly accelerates convergence, reducing training time by 66% compared to benchmarks while maintaining or exceeding detection accuracy. Furthermore, our model achieves 27 Frames Per Second (FPS) on the COCO dataset, setting a new record among DETR-like methods with high detection accuracies.