Hybrid NOMA–MIMO Framework for Maximizing Spectral Efficiency in 6G Networks using Adaptive Beamforming and Dynamic User Clustering

Abstract—The rapid evolution toward sixth-generation (6G)
networks demands innovative physical-layer solutions that
simultaneously maximize spectral efficiency, maintain energy
efficiency, and ensure fairness in ultra-dense deployments. This
paper proposes a Hybrid Non-Orthogonal Multiple Access–
Multiple Input Multiple Output (NOMA–MIMO) framework
integrating adaptive hybrid beamforming and dynamic user
clustering to address these challenges. The system leverages sub-
THz frequencies, massive MIMO arrays, and power-domain
multiplexing to support simultaneous multi-user transmission.
Adaptive hybrid beamforming optimizes analog and digital
precoding stages based on instantaneous channel state
information (CSI), while a graph-based dynamic user clustering
algorithm groups users for optimal NOMA pairing according to
CSI, spatial correlation, and QoS requirements. Power
allocation is optimized using Particle Swarm Optimization
(PSO), balancing fairness and throughput across clusters.
Extensive simulations under 6G channel models demonstrate
the proposed scheme’s superiority over baseline methods such
as OMA–MIMO, fixed-beamforming NOMA, and fully digital
MIMO. The proposed framework achieves a maximum spectral
efficiency of 29.62 bps/Hz for 100 users, an energy efficiency of
14.10 bits/J, a fairness index above 0.97, and a BER reduction
of up to 38% compared to OMA–MIMO. The average accuracy
in user decoding, measured via BER-based reliability, reached
97.85%, confirming robustness under varying user densities
and SNR conditions. The results highlight the Hybrid NOMA–
MIMO framework’s potential as a scalable and energyconscious
access technique for next-generation networks,
offering a balanced trade-off between spectral utilization,
energy consumption, and user fairness. This work establishes a
foundation for integrating advanced access and beamforming
strategies into future 6G deployments.
Keywords—Hybrid NOMA–MIMO, Adaptive Beamforming,
Dynamic User Clustering, Spectral Efficiency, 6G Networks,
Power Allocation Optimization, Sub-Thz Communications,
Fairness Index, Particle Swarm Optimization