Non-orthogonal multiple access (NOMA) is an impressive multi-access technology that enhances the spectral
efficiency of visible light communication (VLC) systems. However, improved spectral efficiency might lead to
performance reduction. To resolve this issue, this paper proposes an innovative approach termed the adaptive
superimposed constellation algorithm (ASCA), which is intended for NOMA-based VLC systems. The proposed
algorithm includes two fundamental methods: the optimal power allocation factor (OPAF) and adaptive M-QAM
selection (AMS). The OPAF method determines the appropriate PA, ensuring non-overlapping decision regions
for superimposed constellation symbols across different modulation orders supported by corresponding theoretical
results. The AMS method dynamically selects the M-QAM modulation order for users, striking a balance
between communication quality and data transmission rates. These methods collectively improve fairness among
users, reduce the symbol error rate (SER), analyze limitations regarding the total number of users the system can
serve, and evaluate the system’s capacity to support multiple users effectively. This paper offers closed-form SER
expressions for served users based on square QAM modulation and an imperfect successive interference
cancellation (SIC) scenario. Theoretical analysis and simulations confirm the findings. The simulation results
show that the proposed algorithm achieves better SER performance than other power domain (PD) NOMA
schemes, making it a reliable and efficient solution for integrating NOMA into VLC systems.