INTELLIGENT SURVEILLANCE FOR WEAPON THREAT DETECTION: TRANSFORMER-BASED DETECTION AND SEGMENTATION USING THERMAL AND RGB IMAGERY

Author

• Akhila Kambhatla, Southern Illinois University CarbondaleFollow

Date of Award

5-1-2026

Degree Name

Doctor of Philosophy

Department

Computer Science

First Advisor

Khaled, Ahmed

Abstract

Proactive public-safety surveillance requires deep learning systems that can detect and segment weapons reliably under real-world conditions. This research develops and evaluates a progressive, multi-phase pipeline that moves (i) from RGB object detection to pixel-level semantic segmentation and (ii) from visible-spectrum cameras to thermal imagery to improve robustness under low-light, clutter, and occlusion. Early phases establish strong detection baselines using one-stage and two-stage detectors (YOLOv5 and Faster R-CNN) for multi-class weapon localization, followed by practical optimization through pruning and ensembling to reduce false positives and computational overhead while maintaining real-time feasibility. Building on the limitations of bounding-box localization, the methodology then transitions to segmentation, enabling fine-grained weapon boundary delineation and improved scene understanding. To support segmentation at scale, large RGB and thermal weapon datasets are constructed and curated, leveraging SAM2-assisted annotation with minimal manual refinement. For RGB multi-class segmentation, a lightweight transformer architecture, ARMFormer, is introduced by integrating attention-driven feature refinement within a hierarchical encoder–decoder design to balance accuracy and edge efficiency. For thermal imagery, transformer-based baselines (SegFormer, SegNeXt, DeepLabV3+, and Swin Transformer) are systematically evaluated for binary weapon segmentation, demonstrating clear accuracy–speed trade-offs under low-texture thermal conditions. Finally, WeaponFormer is proposed as a lightweight transformer model trained from scratch for thermal multi-class segmentation, optimized via spatially efficient attention and compact multi-scale fusion to support real-time threat recognition across handgun, knife, rifle, and human classes. Across experiments, transformer-based segmentation improves robustness and reduces background-induced false predictions compared to detection-only pipelines, and explainable AI analysis is incorporated to interpret model behavior and validate decision consistency. Overall, the work delivers an end-to-end surveillance framework that bridges detection and segmentation across RGB and thermal modalities, alongside datasets, empirical benchmarks, and architecture designs that advance practical weapon threat detection.