CogView: Mastering Text-to-Image Generation via Transformers

• DOI: 10.48550/arxiv.2105.13290
• Publication Date: 2021-01-01
• Authors: Ding, Ming; Yang, Zhuoyi; Hong, Wenyi; Zheng, Wendi; Zhou, Chang; Yin, Da; Lin, Junyang; Zou, Xu; Shao, Zhou; Yang, Hongxia; Tang, Jie
• Keywords: FOS: Computer and information sciences; Computer Science - Machine Learning; Computer Vision and Pattern Recognition (cs.CV); Computer Science - Computer Vision and Pattern Recognition; 0202 electrical engineering, electronic engineering, information engineering; 02 engineering and technology; Machine Learning (cs.LG)

Text-to-Image generation in the general domain has long been an open problem, which requires both a powerful generative model and cross-modal understanding. We propose CogView, a 4-billion-parameter Transformer with VQ-VAE tokenizer to advance this problem. We also demonstrate the finetuning strategies for various downstream tasks, e.g. style learning, super-resolution, text-image ranking and fashion design, and methods to stabilize pretraining, e.g. eliminating NaN losses. CogView achieves the state-of-the-art FID on the blurred MS COCO dataset, outperforming previous GAN-based models and a recent similar work DALL-E.<br/>to appear in NeurIPS 2021<br/>