Journal of AI-Driven Communication Engineering

Special Issue: Artificial Intelligence Across the Communication Stack: Engineering, Human Interaction, and Governance in the 6G Era

Vol. 2 (2026)

Artificial Neural Networks in Next-Generation Communication Systems: Architectures, Applications, and Deployment Challenges

DOI:
https://doi.org/10.31875/2979-1081.2026.02.03
Submitted
July 4, 2026
Published
2026-07-04

Abstract

The proliferation of intelligent wireless systems and the advent of sixth-generation (6G) networks have rendered traditional model-based signal processing approaches increasingly inadequate for managing the complexity, heterogeneity, and dynamism of modern communication infrastructures. Artificial neural networks (ANNs) have emerged as a transformative paradigm, enabling data-driven solutions for tasks ranging from channel estimation and modulation recognition to resource allocation and anomaly detection. This review synthesizes significant developments in ANN architectures applied to communication engineering, spanning multi-layer perceptrons, convolutional and recurrent networks, transformer-based models, graph neural networks, and spiking neural networks, and critically evaluates their applicability within real-world deployment constraints including hardware budgets, latency requirements, and standards compliance. We systematically analyse performance gains across key application domains including adaptive beamforming, end-to-end autoencoder design, federated network management, and energy-efficient edge inference. Furthermore, we examine persistent challenges such as catastrophic forgetting, adversarial vulnerability, data poisoning, model confidentiality risks, interpretability deficits, and the tension between model complexity and real-time deployment. The review concludes by delineating open research directions with emphasis on neuromorphic computing, physics-informed neural networks, and privacy-preserving collaborative learning frameworks.

References

  1. Simeone O: A brief introduction to machine learning for engineers. Foundations and Trends in Signal Processing 2018, 12(3-4): 200-431. https://doi.org/10.1561/2000000102
  2. Saad W, Bennis M, Chen M: A vision of 6G wireless systems: Applications, enabling technologies, and design aspects. IEEE Network 2020, 34(3): 134-142. https://doi.org/10.1109/MNET.001.1900287
  3. Zappone A, Di Renzo M, Debbah M: Wireless networks design in the era of deep learning: Model-based, AI-based, or both? IEEE Transactions on Communications 2019, 67(10): 7331-7376. https://doi.org/10.1109/TCOMM.2019.2924010
  4. Goodfellow I, Bengio Y, Courville A: Deep Learning. MIT Press; 2016.
  5. Huang H, Guo S, Gui G, Yang Z, Zhang J, Sari H, Adachi F: Deep learning for physical-layer 5G wireless techniques: Opportunities, challenges and solutions. IEEE Wireless Communications 2020, 27(1): 214-222. https://doi.org/10.1109/MWC.2019.1900027
  6. O'Shea TJ, West N: Radio machine learning dataset generation with GNU Radio. Proceedings of the GNU Radio Conference; 2016.
  7. Erpek T, O'Shea TJ, Clancy TC: Deep learning for wireless communications: Opportunities, challenges, and potential. IEEE MILCOM; 2018: 1-6.
  8. Kim TH, Yang Z, Zhang J, Li GY: Deep neural network communications over the air. IEEE Transactions on Cognitive Communications and Networking 2020, 6(2): 739-748.
  9. Ye H, Li GY, Juang BH: Power of deep learning for channel estimation and signal detection in OFDM systems. IEEE Wireless Communications Letters 2018, 7(1): 114-117. https://doi.org/10.1109/LWC.2017.2757490
  10. Sun H, Chen X, Shi Q, Hong M, Fu X, Sidiropoulos ND: Learning to optimize: Training deep neural networks for interference management. IEEE Transactions on Signal Processing 2018, 66(20): 5438-5453. https://doi.org/10.1109/TSP.2018.2866382
  11. O'Shea TJ, Hoydis J: An introduction to deep learning for the physical layer. IEEE Transactions on Cognitive Communications and Networking 2017, 3(4): 563-575. https://doi.org/10.1109/TCCN.2017.2758370
  12. Wang X, Han Y, Leung VCM, Niyato D, Yan X, Chen X: Convergence of edge computing and deep learning: A comprehensive survey. IEEE Communications Surveys and Tutorials 2020, 22(2): 869-904. https://doi.org/10.1109/COMST.2020.2970550
  13. Hornik K, Stinchcombe M, White H: Multilayer feedforward networks are universal approximators. Neural Networks 1989, 2(5): 359-366. https://doi.org/10.1016/0893-6080(89)90020-8
  14. Liu Y, Pan W, Shao T, Tang Y: A new digital predistortion for wideband power amplifiers with crosstalk in MIMO transmitters. IEEE Microwave and Wireless Components Letters 2016, 26(3): 218-220.
  15. O'Shea TJ, Roy T, Clancy TC: Over-the-air deep learning based radio signal classification. IEEE Journal of Selected Topics in Signal Processing 2018, 12(1): 168-179. https://doi.org/10.1109/JSTSP.2018.2797022
  16. Hochreiter S, Schmidhuber J: Long short-term memory. Neural Computation 1997, 9(8): 1735-1780. https://doi.org/10.1162/neco.1997.9.8.1735
  17. Soltani M, Pourahmadi V, Mirzaei A, Sheikhzadeh H: Deep learning-based channel estimation. IEEE Communications Letters 2019, 23(4): 652-655. https://doi.org/10.1109/LCOMM.2019.2898944
  18. Liao Y, Yao H, Hua Y, Li C: CSI feedback based on deep learning for massive MIMO systems. IEEE Access 2019, 7: 86810-86820. https://doi.org/10.1109/ACCESS.2019.2924673
  19. Vaswani A, Shazeer N, Parmar N, et al.: Attention is all you need. Advances in Neural Information Processing Systems 2017, 30: 5998-6008.
  20. He H, Wen CK, Jin S, Li GY: A model-driven deep learning network for MIMO detection. IEEE GlobalSIP; 2018: 584-588. https://doi.org/10.1109/GlobalSIP.2018.8646357
  21. Dong P, Zhang H, Li GY, Gaspar IS, NaderiAlizadeh N: Deep CNN-based channel estimation for mmWave massive MIMO systems. IEEE Journal of Selected Topics in Signal Processing 2019, 13(5): 989-1000. https://doi.org/10.1109/JSTSP.2019.2925975
  22. Scarselli F, Gori M, Tsoi AC, Hagenbuchner M, Monfardini G: The graph neural network model. IEEE Transactions on Neural Networks 2009, 20(1): 61-80. https://doi.org/10.1109/TNN.2008.2005605
  23. Shen Y, Shi Y, Zhang J, Letaief KB: Graph neural networks for scalable radio resource management: Architecture design and theoretical analysis. IEEE Journal on Selected Areas in Communications 2021, 39(1): 101-115. https://doi.org/10.1109/JSAC.2020.3036965
  24. Eisen M, Ribeiro A: Optimal wireless resource allocation with random edge graph neural networks. IEEE Transactions on Signal Processing 2020, 68: 2977-2991. https://doi.org/10.1109/TSP.2020.2988255
  25. Maass W: Networks of spiking neurons: The third generation of neural network models. Neural Networks 1997, 10(9): 1659-1671. https://doi.org/10.1016/S0893-6080(97)00011-7
  26. Davies M, Srinivasa N, Lin TH, et al.: Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro 2018, 38(1): 82-99. https://doi.org/10.1109/MM.2018.112130359
  27. Kim Y, Park J, Choi S, et al.: Efficient spiking neural networks for IoT sensor anomaly detection. IEEE Internet of Things Journal 2024, 11(4): 6231-6243.
  28. Haykin S: Adaptive Filter Theory, 5th ed. Pearson; 2013.
  29. Widrow B, Stearns SD: Adaptive Signal Processing. Prentice-Hall; 1985.
  30. Neumann D, Wiese T, Utschick W: Learning the MMSE channel estimator. IEEE Transactions on Signal Processing 2018, 66(11): 2905-2917. https://doi.org/10.1109/TSP.2018.2799164
  31. Yang Y, Gao F, Zhong Z, Ai B, Alkhateeb A: Deep transfer learning-based downlink channel prediction for FDD massive MIMO systems. IEEE Transactions on Communications 2020, 68(12): 7485-7497. https://doi.org/10.1109/TCOMM.2020.3019077
  32. West NE, O'Shea TJ: Deep architectures for modulation recognition. IEEE DySPAN; 2017: 1-6. https://doi.org/10.1109/DySPAN.2017.7920754
  33. Liu X, Yang D, El Gamal A: Deep neural network architectures for modulation classification. 51st Asilomar Conference; 2017: 915-919. https://doi.org/10.1109/ACSSC.2017.8335483
  34. Jaraut P, Rawat M, Ghannouchi FM: Composite neural network digital predistortion model. IEEE Transactions on Microwave Theory and Techniques 2018, 66(5): 2246-2255.
  35. Liu C, Fang B, Miao H, et al.: Neural network digital predistortion for wideband RF power amplifiers. IEEE Microwave and Wireless Technology Letters 2023, 33(7): 1027-1030. https://doi.org/10.1109/LMWT.2023.3322273
  36. Sun Y, Ng DWK, Ding Z, Schober R: Optimal joint power and subcarrier allocation for full-duplex multicarrier NOMA systems. IEEE Transactions on Communications 2017, 65(3): 1077-1091. https://doi.org/10.1109/TCOMM.2017.2650992
  37. Luong NC, Hoang DT, Gong S, Niyato D, Wang P, Liang YC, Kim DI: Applications of deep reinforcement learning in communications and networking. IEEE Communications Surveys and Tutorials 2019, 21(4): 3133-3174. https://doi.org/10.1109/COMST.2019.2916583
  38. Lin T, Zhu Y: Beamforming design for large-scale antenna arrays using deep learning. IEEE Wireless Communications Letters 2020, 9(1): 103-107. https://doi.org/10.1109/LWC.2019.2943466
  39. Xia W, Zheng G, Zhu Y, Zhang J, Wang J, Quek TQS: A deep learning framework for optimization of MISO downlink beamforming. IEEE Transactions on Communications 2020, 68(3): 1866-1880. https://doi.org/10.1109/TCOMM.2019.2960361
  40. Mnih V, Kavukcuoglu K, Silver D, et al.: Human-level control through deep reinforcement learning. Nature 2015, 518: 529-533. https://doi.org/10.1038/nature14236
  41. Wang Z, Wang J: Deep reinforcement learning for dynamic multichannel access in wireless networks. IEEE Transactions on Cognitive Communications and Networking 2018, 4(2): 257-265. https://doi.org/10.1109/TCCN.2018.2809722
  42. Naparstek O, Cohen K: Deep multi-user reinforcement learning for distributed dynamic spectrum access. IEEE Transactions on Wireless Communications 2019, 18(1): 310-323. https://doi.org/10.1109/TWC.2018.2879433
  43. Foerster JN, Assael YM, de Freitas N, Whiteson S: Learning to communicate with deep multi-agent reinforcement learning. Advances in Neural Information Processing Systems 2016, 29: 2137-2145.
  44. Shi Y, Zhang J, Letaief KB, Bai B, Chen W: Large-scale convex optimization for ultra-dense cloud-RAN. IEEE Wireless Communications 2015, 22(3): 84-91. https://doi.org/10.1109/MWC.2015.7143330
  45. He Z, Zhang J, Shi Y, Letaief KB: GNN-based distributed resource allocation for 6G networks. IEEE Journal on Selected Areas in Communications 2023, 41(6): 1818-1832.
  46. Gama F, Marques AG, Leus G, Ribeiro A: Convolutional neural network architectures for signals supported on graphs. IEEE Transactions on Signal Processing 2019, 67(4): 1034-1049. https://doi.org/10.1109/TSP.2018.2887403
  47. Proakis JG, Salehi M: Digital Communications, 5th ed. McGraw-Hill; 2008.
  48. Dorner S, Cammerer S, Hoydis J, ten Brink S: Deep learning-based communication over the air. IEEE Journal of Selected Topics in Signal Processing 2018, 12(1): 132-143. https://doi.org/10.1109/JSTSP.2017.2784180
  49. O'Shea TJ, Hoydis J: An introduction to deep learning for the physical layer. IEEE Transactions on Cognitive Communications and Networking 2017, 3(4): 563-575. https://doi.org/10.1109/TCCN.2017.2758370
  50. Cammerer S, Gruber T, Hoydis J, ten Brink S: Trainable communication systems: Concepts and prototype. IEEE Transactions on Communications 2020, 68(9): 5489-5503. https://doi.org/10.1109/TCOMM.2020.3002915
  51. Aoudia FA, Hoydis J: End-to-end learning of communications systems without a channel model. 52nd Asilomar Conference; 2018: 298-303. https://doi.org/10.1109/ACSSC.2018.8645416
  52. Ye H, Li GY, Juang BH, Sivanesan K: Channel agnostic end-to-end learning based communication systems with conditional GAN. IEEE Globecom; 2018: 1-5. https://doi.org/10.1109/GLOCOMW.2018.8644250
  53. Kingma DP, Welling M: Auto-encoding variational Bayes. ICLR; 2014.
  54. Jiang W, Schotten HD: Neural network-based fading channel prediction: A comprehensive overview. IEEE Access 2019, 7: 118112-118124. https://doi.org/10.1109/ACCESS.2019.2937588
  55. Guo H, Turel O, Li Y, Chen W: Transformer-based end-to-end multi-user MIMO system. IEEE Wireless Communications Letters 2023, 12(5): 858-862.
  56. Mao Y, You C, Zhang J, Huang K, Letaief KB: A survey on mobile edge computing: The communication perspective. IEEE Communications Surveys and Tutorials 2017, 19(4): 2322-2358. https://doi.org/10.1109/COMST.2017.2745201
  57. Han S, Pool J, Tran J, Dally W: Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems 2015, 28: 1135-1143.
  58. Kim S, Moon I, Lee D, et al.: Integer-only quantization for efficient DNN-based channel estimation. IEEE Wireless Communications Letters 2023, 12(3): 500-504.
  59. Pfeiffer M, Pfeil T: Deep learning with spiking neurons: Opportunities and challenges. Frontiers in Computational Neuroscience 2018, 12: 88. https://doi.org/10.3389/fnins.2018.00774
  60. Orchard G, Jayawant A, Cohen GK, Thakor N: Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in Neuroscience 2015, 9: 437. https://doi.org/10.3389/fnins.2015.00437
  61. Shi L, Li G, Yuan M, et al.: Neuromorphic edge computing for IoT anomaly detection. IEEE Transactions on Circuits and Systems I 2024, 71(2): 733-746.
  62. McMahan HB, Moore E, Ramage D, Hampson S, Arcas BA: Communication-efficient learning of deep networks from decentralized data. AISTATS; 2017: 1273-1282.
  63. Niknam S, Dhillon HS, Reed JH: Federated learning for wireless communications: Motivation, opportunities, and challenges. IEEE Communications Magazine 2020, 58(6): 46-51. https://doi.org/10.1109/MCOM.001.1900461
  64. Geyer RC, Klein T, Nabi M: Differentially private federated learning: A client level perspective. arXiv:1712.07557; 2017.
  65. Elbir AM, Soner B, Coleri S: Federated learning in vehicular networks. IEEE PIMRC; 2020: 1-6.
  66. McCloskey M, Cohen NJ: Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation 1989, 24: 109-165. https://doi.org/10.1016/S0079-7421(08)60536-8
  67. Kirkpatrick J, Pascanu R, Rabinowitz N, et al.: Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 2017, 114(13): 3521-3526. https://doi.org/10.1073/pnas.1611835114
  68. Szegedy C, Zaremba W, Sutskever I, et al.: Intriguing properties of neural networks. ICLR; 2014.
  69. Flowers B, Buehrer RM, Headley WC: Evaluating adversarial evasion attacks in the context of wireless communications. IEEE Transactions on Information Forensics and Security 2019, 14(11): 2891-2900.
  70. Madry A, Makelov A, Schmidt L, Tsipras D, Vladu A: Towards deep learning models resistant to adversarial attacks. ICLR; 2018.
  71. Raissi M, Perdikaris P, Karniadakis GE: Physics-informed neural networks: A deep learning framework for solving forward and inverse problems. Journal of Computational Physics 2019, 378: 686-707. https://doi.org/10.1016/j.jcp.2018.10.045
  72. Jiang R, Chen X, Zhong C, Zhang Z: Physics-inspired deep learning for channel estimation in massive MIMO. IEEE Communications Letters 2023, 27(1): 173-177.
  73. Sze V, Chen YH, Yang TJ, Emer JS: Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE 2017, 105(12): 2295-2329. https://doi.org/10.1109/JPROC.2017.2761740
  74. Sebastian A, Le Gallo M, Khaddam-Aljameh R, Eleftheriou E: Memory devices and applications for in-memory computing. Nature Nanotechnology 2020, 15(7): 529-544. https://doi.org/10.1038/s41565-020-0655-z
  75. Ambrogio S, Narayanan P, Tsai H, et al.: Equivalent-accuracy accelerated neural-network training using analogue memory. Nature 2018, 558(7708): 60-67. https://doi.org/10.1038/s41586-018-0180-5