Automated detection of small objects poses additional challenges, compared to bigger-sized ones, due to the former’s limited resolution for extracting discriminative information. In such cases, even a slight misalignment between a candidate region and its ground truth target has a huge impact on their IoU which significantly increases the amount of noisy information. Given the fact that state-of-the-art two-stage detection algorithms generate predefined shaped and sized candidate regions in pixel-level interval, the aforementioned misalignments are very likely to occur. In this work, a scalable object detection approach is introduced- specifically dedicated to small object parts- incorporating both learnable and handcrafted features . In particular, a set of simplified Gabor waveforms (SGWs) is applied to the raw data, ultimately producing an improved set of anchors for the region proposal network. These Gabor filters are further utilized generating a soft attention mask. Additionally, the interaction of a human with the object is also exploited by taking advantage of affordance-based information for further improvement of detection performance. Experiments have been conducted in a newly introduced device disassembly segmentation dataset, demonstrating the robustness of the method in detection of small device components.