Parveen, NagmaNagmaParveen2026-01-202026-01-202025https://studentrepo.iium.edu.my/handle/123456789/33785The coexistence of 5G networks and satellite communication systems within the C-band (3.4–4.2 GHz) has created significant challenges due to overlapping frequency usage and the resulting potential for harmful interference. As both terrestrial and satellite services increasingly depend on this spectrum for high-capacity communication, ensuring interference-free operation has become essential for maintaining reliable connectivity. This shared spectrum environment demands accurate interference detection and robust mitigation strategies to safeguard system performance. Addressing this need, the present research develops an advanced deep learning–based interference mitigation framework using convolutional neural networks (CNN) to model, classify, and suppress interference in mixed 5G–satellite environments. Comprehensive analysis of 5G and satellite datasets provided insights into their spectral characteristics and interference behavior, enabling effective feature extraction and supporting the development of the proposed technique. Modulation classification using a CNN achieved an accuracy of 97.83%, demonstrating strong capability in identifying modulation types under diverse interference conditions. For interference classification, the CNN achieved 98% accuracy, successfully distinguishing satellite-only, wireless-only, and combined interference scenarios. To further enhance system performance, optimization was performed using a feedforward neural network integrated with a genetic algorithm (GA). This optimization process adjusted key communication parameters, including the beam-forming angle, transmit power, and interference threshold, resulting in improved spatial filtering and interference suppression. The optimized beamforming direction of approximately 21° corresponded to the most favorable steering angle for isolating the desired signal from interference sources. This resulted in a substantial SINR improvement of +18 dB, equivalent to a 55% improvement in overall signal quality. Additionally, the regression-based prediction of the beamforming angle achieved an overall R-value of 0.975, confirming the high accuracy and reliability of the predictive model. Benchmark validation demonstrated strong improvements in modulation recognition performance. The proposed CNN achieved an accuracy of 97.83%, outperforming other deep-learning models reported in the literature, which typically achieve 87–91% under similar channel conditions. Earlier studies on higher-order modulation schemes showed performance ranging from 60% to 95%, particularly degrading under low-SNR and multipath scenarios. Compared to these ranges, the proposed CNN demonstrated the highest and most stable accuracy across all tested conditions. Furthermore, the simulated interference classification accuracy of 98% closely matched the theoretical confusion matrix, with less than 2% deviation, confirming the robustness and analytical consistency of the proposed framework. Overall, this research presents a practical and effective methodology for interference detection, classification, and mitigation within the C-band. The proposed framework enhances signal integrity, improves spectral efficiency, and supports dependable 5G–satellite coexistence, contributing to the advancement of future intelligent wireless communication systems.enJOINTLY OWNED WITH A THIRD PARTY(S) AND/OR IIUMCoexistence;Satellite;5GWireless communication systems -- Technological innovationsDeep learningDoctoral Theses