Virtual speaker generation is an important research direction in the field of artificial intelligence, aiming to generate virtual speakers with realistic voices through computers. However, existing methods often neglect emotional expression and the facial details of the generated face images lack realism, limiting the performance and interactivity of the virtual speakers. To address this issue, a Transformer-based Generative Adversarial Network (GAN) method is proposed for generating virtual speakers with different emotions(GANLTB) . This method is based on the GAN network architecture, where the generator uses a Transformer model to process speech and image features, combined with emotional condition information and latent space vectors, to generate voice and images with specified emotions. The discriminator is used to assess the authenticity of the generated results and provide gradient signals to guide the training of the generator. By introducing bicubic interpolation, the image quality of the virtual speaker generation is further enhanced, making the facial details of the virtual speaker clearer and the expressions more natural and vivid. The method was validated using the CREMA-D dataset, a diverse emotional dataset, through subjective evaluation and objective indicators to assess the emotional expression ability and quality of the generated speech and images. Experimental results show that the method can generate virtual speakers with diverse and realistic emotional expressions. Compared to other advanced methods currently available, the proposed method is clearer in details such as fluency and realism, bringing a better sense of reality.

[2024-01-24] Accepted Paper /doc/2023.10.0559/v1