Reinforcement learning-inspired molecular generation with latent space diffusion and genetic algorithm optimization under affinity and similarity constraints - Bai, Can; Wu, Zijian; Han, Xianjun; Huang, Renbao (Chemical engineering science, 2026)
•A novel RL-inspired framework combines VAE and latent-space diffusion for molecule generation.•Affinity and similarity constraints guide generation toward biologically active candidates.•A genetic algorithm with active learning enables iterative, reward-driven optimization. In deep learning-driven molecular generation, achieving both diversity and effectiveness remains a major challenge. Inspired by the principles of reinforcement learning, in this study, a molecular generation system that integrates an encoding–diffusion–decoding mechanism with iterative, feedback-driven optimization strategies to balance these two aspects is proposed. The framework first maps molecular structures into a low-dimensional latent space, where a diffusion model explores the distribution of molecular characteristics. Sampling from a Gaussian distribution and performing reverse decoding ensure diversity in the molecular generation process. To ensure the practical applicability of the generated molecules, we incorporate a target-drug affinity prediction model and molecular similarity constraints into the pipeline to effectively filter candidates that are both novel and biologically relevant. Furthermore, a molecular genetic algorithm that mimics the exploration and exploitation trade-off that is fundamental to reinforcement learning is employed to perform random crossover and mutation on selected molecules, thereby generating potentially superior candidates. Guided by an active learning strategy, these candidates are iteratively evaluated and integrated into the training set, thus forming a continuous feedback loop that refines the generation model over time. This reinforcement learning-inspired framework not only increases the quality and efficiency of molecular generation but also reduces dependency on large, high-quality datasets. The experimental results confirm the ability of the method to generate effective and diverse compounds that target specific receptors.