Skip to main navigation menu Skip to main content Skip to site footer
International Journal of Robotics and Automation Technology

Articles

Vol. 10 (2023)

Research on Semantic Segmentation of Fish-Eye Images for Autonomous Driving

DOI
https://doi.org/10.31875/2409-9694.2023.10.13
Submitted
December 27, 2023
Published
27.12.2023

Abstract

Abstract: Fisheye cameras, valued for their wide field of view, play a crucial role in perceiving the surrounding environment of vehicles. However, there is a lack of specific research addressing the processing of significant distortion features in segmenting fish-eye images. Additionally, fish-eye images for autonomous driving face the challenge of few datasets, potentially causing over fitting and hindering the model's generalization ability.

Based on the semantic segmentation task, a method for transforming normal images into fish-eye images is proposed, which expands the fish-eye image dataset. By employing the Transformer network and the Across Feature Map Attention, the segmentation performance is further improved, achieving a 55.6% mIOU on Woodscape. Additionally, leveraging the concept of knowledge distillation, the network ensures a strong generalization based on dual-domain learning without compromising performance on Woodscape (54% mIOU).

References

  1. Kumar, Varun Ravi, et al. “Surround-View Fisheye Camera Perception for Automated Driving: Overview, Survey & Challenges.” IEEE Transactions on Intelligent Transportation Systems 24 (2022): 3638-3659. https://doi.org/10.1109/TITS.2023.3235057
  2. Ekkat, Ahmed Rida et al. “SynWoodScape: Synthetic Surround-View Fisheye Camera Dataset for Autonomous Driving.” IEEE Robotics and Automation Letters 7 (2022): 8502-8509. https://doi.org/10.48550/arXiv.2203.05056
  3. Shelhamer, Evan et al. “Fully convolutional networks for semantic segmentation.” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014): 3431-3440. https://doi.org/10.48550/arXiv.1411.4038
  4. Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer International Publishing, 2015. https://doi.org/10.48550/arXiv.1505.04597
  5. Zhao, Hengshuang et al. “Pyramid Scene Parsing Network.” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016): 6230-6239. https://doi.org/10.48550/arXiv.1612.01105
  6. Wang, Wenhai et al. “Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions.” 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (2021): 548-558. https://doi.org/10.48550/arXiv.2102.12122
  7. Xie, Enze et al. “SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers.” Neural Information Processing Systems (2021). https://doi.org/10.48550/arXiv.2105.15203
  8. Gu et al. “Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation.” 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021): 12084-12093. https://doi.org/10.48550/arXiv.2111.01236
  9. Hänisch, Evangelio, Tadjine and Pätzold, "Free-space detection with fish-eye cameras," 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 135-140. http://doi.org/10.1109/IVS.2017.7995710
  10. Deng, Yang, Qian et al. "CNN based semantic segmentation for urban traffic scenes using fisheye camera," 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 231-236. http://doi.org/10.1109/IVS.2017.7995725
  11. Sáez, Bergasa, Romeral et al. "CNN-based Fisheye Image Real-Time Semantic Segmentation," 2018 IEEE Intelligent Vehicles Symposium (IV), 2018, pp. 1039-1044. http://doi.org/10.1109/IVS.2018.8500456
  12. Blott, G., Takami, M., & Heipke, C. (2018). Semantic Segmentation of Fisheye Images. ECCV Workshops. https://doi.org/10.1007/978-3-030-11009-3_10
  13. Deng, Liuyuan et al. “Restricted Deformable Convolution-Based Road Scene Semantic Segmentation Using Surround View Cameras.” IEEE Transactions on Intelligent Transportation Systems 21 (2018): 4350-4362. http://doi.org/10.1109/TITS.2019.2939832
  14. Ye, Yaozu et al. “Universal Semantic Segmentation for Fisheye Urban Driving Images.” 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (2020): 648-655. https://doi.org/10.1109/SMC42975.2020.9283099
  15. Sang, Zhou, Islam and Xing, "Small-Object Sensitive Segmentation Using Across Feature Map Attention," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 5, pp. 6289-6306, 1 May 2023. https://doi.org/10.1109/TPAMI.2022.3211171
  16. Hinton, Vinyals & Dean (2015). Distilling the Knowledge in a Neural Network. ArXiv, abs/1503.02531. https://doi.org/10.48550/arXiv.1503.02531
  17. Romero, Ballas, Kahou, Chassang, Gatta, & Bengio. (2014). FitNets: Hints for Thin Deep Nets. CoRR, abs/1412.6550. https://doi.org/10.48550/arXiv.1412.6550
  18. Miyamoto, “Fish eye lens,” Journal of the Optical Society of America, vol. 54, no. 8, pp. 1060-1061, 1964.
  19. Ramachandran, Saravanabalagi et al. “Woodscape Fisheye Semantic Segmentation for Autonomous Driving - CVPR 2021 OmniCV Workshop Challenge.” ArXiv abs/2107.08246 (2021). https://doi.org/10.48550/arXiv.2107.08246
  20. Yogamani, Senthil Kumar et al. “Woodscape: A Multi-Task, Multi-Camera Fisheye Dataset for Autonomous Driving.” 2019 IEEE/CVF International Conference on Computer Vision (ICCV): 9307-9317. https://doi.org/10.1109/ICCV.2019.00940