| 154 | 0 | 188 |
| 下载次数 | 被引频次 | 阅读次数 |
内河无人艇在环境监测、水域救援、交通运输等领域展现出重要应用价值，但其感知系统面临水面反射、恶劣光照及多变天气等严峻挑战。目前水面感知研究主要依赖于摄像头或低分辨率毫米波雷达数据，难以满足复杂场景下的多模态感知需求。针对这一挑战，提出一种基于4D毫米波雷达-摄像头融合的解决方案：首先，构建4D毫米波雷达的多维特征表征体系，包括动态提取距离、方位、速度和反射强度等关键特征；然后，设计动态场景自适应的跨模态融合机制，利用注意力机制加权融合不同模态特征，实现实时自适应算法以应对环境变化；最后，通过深度学习模型实现异构传感器在特征层级的深度耦合。实验验证结果表明：该融合方案显著提升了感知系统的环境适应性，在低质量光照条件下和恶劣天气场景中，目标检测精度较传统视觉系统分别提升3.4和3.9个百分点。本研究不仅为水面复杂场景的智能感知提供了有效的技术解决方案，还将进一步推动内河无人艇向智能化、自主化方向发展。
Abstract：Inland unmanned vessels demonstrate significant application value in environmental monitoring, water rescue, and transportation. However, their perception systems face critical challenges including water surface reflections,adverse illumination, and variable weather conditions. Current aquatic perception research primarily relies on camera or low-resolution radar data, which fails to meet multimodal sensing requirements in complex scenarios. To address these challenges, this research proposes a 4D radar-camera fusion solution. We first construct a multidimensional feature representation system for 4D radar that dynamically extracts key features including distance, azimuth, velocity,and reflection intensity. Subsequently, we design a dynamic scene-adaptive cross-modal fusion mechanism that employs attention-based weighting to effectively integrate different modal features, enabling real-time adaptive algorithms to handle environmental variations. The heterogeneous sensors are then deeply coupled at the feature level through a carefully designed deep learning model. Experimental validation demonstrates significant improvements in environmental adaptability, with the proposed fusion solution achieving target detection accuracy improvements of3.4 and 3.9 percentage points over traditional vision systems under poor lighting and harsh weather conditions,respectively. This research not only provides an effective technical solution for intelligent perception in complex aquatic environments, but also advances the development of inland unmanned vessels toward intelligent and autonomous operation.
[1] FENG D, HAASE-SCHüTZ C, ROSEN BAUM L, et al.Deep multi-modal object detection and semantic segmentation for autonomous driving:datasets, methods, and challenges[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(3):1341-1360.
[2] WU X, LI W, HONG D F, et al. Deep learning for unmanned aerial vehicle-based object detection and tracking:a survey[J]. IEEE Geoscience and Remote Sensing Magazine, 2022, 10(1):91-124.
[3] BOVCON B, MUHOVIC J, VRANAC D, et al. MODS:a USV-oriented object detection and obstacle segmentation benchmark[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(8):13403-13418.
[4] BOVCON B , MUHOVIC J , PERS J , et al. The MaSTr1325 dataset for training deep USV obstacle detection models[C]//Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS),November 3-8 , 2019 , Macao , China. New York:IEEE Xplore, 2020:3431-3438.
[5] CHENG Y W, XU H, LIU Y M. Robust small object detection on the water surface through fusion of camera and millimeter wave radar[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision(ICCV), October 10-17, 2021, Montreal, QC, Canada. New York:IEEE Xplore, 2022:15243-15252.
[6] LIN J Y, DIEKMANN P, FRAMING C E, et al. Maritime environment perception based on deep learning[J]. IEEE Transactions on Intelligent Transportation Systems, 2022,23(9):15487-15497.
[7] WANG N, CHEN T K, LIU S M, et al. Deep learningbased visual detection of marine organisms:a survey[J].Neurocomputing, 2023, 532:1-32.
[8] WANG N, HE H K, HOU Y L, et al. Model-free visual servo swarming of manned-unmanned surface vehicles with visibility maintenance and collision avoidance[J]. IEEE Transactions on Intelligent Transportation Systems, 2024,25(1):697-709.
[9] HE H K, WANG N, HUANG D Z, et al. Active visionbased finite-time trajectory-tracking control of an unmanned surface vehicle without direct position measurements[J]. IEEE Transactions on Intelligent Transportation Systems, 2024, 25(9):12151-12162.
[10] FANG S M, LIU Z J, WANG X J, et al. Dynamic analysis of emergency evacuation in a rolling passenger ship using a two-layer social force model[J]. Expert Systems with Applications, 2024, 247:123310.
[11] WANG N, WANG Y Y, ER M J. Review on deep learning techniques for marine object recognition:Architectures and algorithms[J]. Control Engineering Practice , 2022, 118:104458.
[12] WANG N, WANG Y Y, FENG Y, et al. AodeMar:attention-aware occlusion detection of vessels for maritime autonomous surface ships[J]. IEEE Transactions on Intelligent Transportation Systems, 2024, 25(10):13584-13597.
[13] LIU R W, LU Y X, GUO Y, et al. AiOENet:all-in-one low-visibility enhancement to improve visual perception for intelligent marine vehicles under severe weather conditions[J]. IEEE Transactions on Intelligent Vehicles, 2024, 9(2):3811-3826.
[14] QU J X, LIU R W, GAO Y, et al. Double domain guided real-time low-light image enhancement for ultra-high-definition transportation surveillance[J]. IEEE Transactions on Intelligent Transportation Systems, 2024, 25(8):9550-9562.
[15] WANG N, WANG Y Y, FENG Y, et al. MDD-ShipNet:math-data integrated defogging for fog-occlusion ship detection[J]. IEEE Transactions on Intelligent Transportation Systems, 2024, 25(10):15040-15052.
[16] XIE B Q, YANG Z M, YANG L, et al. AMMF:attentionbased multi-phase multi-task fusion for small contour object 3D detection[J]. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(2):1692-1701.
[17] LI Y J, PARK J, O′TOOLE M, et al. Modality-agnostic learning for radar-lidar fusion in vehicle detection[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), June 18-24, 2022,New Orleans, LA, USA. New York:IEEE Xplore, 2022:908-917.
[18] LIU Z X, ZHANG Y M, YU X, et al. Unmanned surface vehicles:an overview of developments and challenges[J].Annual Reviews in Control, 2016, 41:71-93.
[19] CHENG Y W, JIANG M X, ZHU J N, et al. Are we ready for unmanned surface vehicles in inland waterways?the USVInland multisensor dataset and benchmark[J].IEEE Robotics and Automation Letters, 2021, 6(2):3964-3970.
[20] OUAKNINE A, NEWSON A, REBUT J, et al. CARRADA dataset:camera and automotive radar with range-angle-Doppler annotations[C]//Proceedings of the 2020 25th International Conference on Pattern Recognition(ICPR),January 10-15, 2021, Milan, Italy. New York:IEEE Xplore,2021:5068-5075.
[21] BIJELIC M , GRUBER T, MANNAN F, et al. Seeing through fog without seeing fog:deep multimodal sensor fusion in unseen adverse weather[C]//Proceedings of the 2020IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), June 13-19, 2020, Seattle, WA, USA.New York:IEEE Xplore, 2020:11679-11689.
[22] YAO S L, GUAN R W, PENG Z T, et al. Exploring radar data representations in autonomous driving:a comprehensive review[J]. IEEE Transactions on Intelligent Transportation Systems, 2025, 26(6):7401-7425.
[23] BILIK I. Comparative analysis of radar and lidar technologies for automotive applications[J]. IEEE Intelligent Transportation Systems Magazine, 2023, 15(1):244-269.
[24] SCHEINER N, KRAUS F, WEI F Y, et al. Seeing around street corners:non-line-of-sight detection and tracking inthe-wild using Doppler radar[C]//Proceedings of the 2020IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), June 13-19, 2020, Seattle, WA, USA.New York:IEEE Xplore, 2020:2065-2074.
[25] WANG Y Z, JIANG Z Y, LI Y D, et al. RODNet:a realtime radar object detection network cross-supervised by camera-radar fused object 3D localization[J]. IEEE Journal of Selected Topics in Signal Processing, 2021, 15(4):954-967.
[26] LIU J N, XIONG W Y, BAI L P, et al. Deep instance segmentation with automotive radar detection points[J].IEEE Transactions on Intelligent Vehicles, 2023, 8(1):84-94.
[27] CHENG Y W, ZHU J N, JIANG M X, et al. FloW:a dataset and benchmark for floating waste detection in inland waters[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision(ICCV), October10-17, 2021, Montreal, QC, Canada. New York:IEEE Xplore, 2022:10933-10942.
[28] YAO S L, GUAN R W, HUANG X Y, et al. Radar-camera fusion for object detection and semantic segmentation in autonomous driving:a comprehensive review[J]. IEEE Transactions on Intelligent Vehicles, 2024, 9(1):2094-2128.
[29] YAO S L, GUAN R W, NI Y, et al. USVTrack:USVbased 4D radar-camera tracking dataset for autonomous driving in inland waterways[EB/OL].(2025-06-23)[2025-07-20]. https：//arxiv.org/html/2506.18737v1.
[30] GE Z, LIU S T, WANG F, et al. YOLOX:exceeding YOLO series in 2021[EB/OL].(2021-07-18)[2025-05-20].https：//arxiv.org/abs/2107.08430.
[31] YAO S L, GUAN R W, WU Z D, et al. WaterScenes:a multi-task 4D radar-camera fusion dataset and benchmarks for autonomous driving on water surfaces[J]. IEEE Transactions on Intelligent Transportation Systems, 2024,25(11):16584-16598.
[32] GIRSHICK R. Fast R-CNN[C]//Proceedings of the 2015IEEE International Conference on Computer Vision(ICCV), December 7-13, 2015, Santiago, Chile. New York:IEEE Xplore, 2016:1440-1448.
[33] ZHOU X Y, WANG D Q, KR?HENBüHL P. Objects as points[EB/OL].(2019-04-16)[2025-05-20]. https：//arxiv.org/abs/1904.07850.
[34] ZHU X Z, SU W J, LU L W, et al. Deformable DETR:deformable transformers for end-to-end object detection[EB/OL].(2020-08-08)[2025-05-20]. https：//arxiv.org/abs/2010.04159.
[35] JOCHER G, CHAURASIA A, QIU J. YOLO by ultralytics[EB/OL][2025-05-20]. https：//github.com/ultralytics/ultralytics.
[36]王海鹏，丁卫平，黄嘉爽，等. FTransCNN：基于模糊融合的Transformer-CNN不确定性医学图像分割模型[J].小型微型计算机系统，2024, 45(6):1426-1435.WANG H P, DING W P, HUANG J S, et al. FTransCNN:fusing transformer and CNN based on fuzzy fusion for uncertain medical image segmentation[J]. Journal of Chinese Computer Systems , 2024, 45(6):1426-1435.(in Chinese)
[37]王静，丁卫平，尹涛，等.基于多模态模糊特征融合的脑龄协同预测算法[J].模式识别与人工智能，2024, 37(7):613-625.WANG J, DING W P, YIN T, et al. Collaborative brain age prediction algorithm based on multimodal fuzzy feature fusion[J]. Pattern Recognition and Artificial Intelligence ,2024, 37(7):613-625.(in Chinese)
基本信息:
DOI：10.12194/j.ntu.20250620001
中图分类号:U664.82;TN958
引用信息:
[1]姚善良,管润玮,丁卫平,等.面向内河无人艇智能航行的4D毫米波雷达-摄像头融合算法[J].南通大学学报(自然科学版),2025,24(03):44-51.DOI:10.12194/j.ntu.20250620001.
基金信息: