High-Capacity Coherent WDM Networks Empowered by Probabilistic Shaping and End-to-End Deep Learning

To optimize the functionality of coherent optical fiber communication (OFC) systems and enhance their capacity related to long-haul transmissions, wavelength-division multiplexing (WDM) and probabilistic constellation shaping (PCS) techniques have been used. This paper develops an end-to-end (E2E) deep learning (DL)-based PCS algorithm, i.e., autoencoder (AE) for a high-order modulation format WDM system that minimizes nonlinear effects while ensuring high capacity and considers system parameters, in particular those related to the OFC channel. Only the AE of the central channel is trained to meet the specified performance objective, as the system design employs identical AEs in each WDM channel. The simulation results show that the architecture should consist of two hidden layers, with thirty two nodes per hidden layer and a "32 × modulation order" batch size to obtain optimal system performance, when designing AE using a dense layer neural network. The behavior of the AE is examined to determine the optimum launch-power ranges that enhance the system's performance. The developed AE-based PCS-WDM provides a 0.4 shaping gain and outperforms conventional solutions.