Publications

♠ Mentored Intern/Student

Agent-DocEdit: Language-Instructed LLM Agent for Content-Rich Document Editing

Te-Lin Wu, Rajiv Jain, Yufan Zhou, Puneet Mathur, Vlad I Morariu.

First Conference on Language Modeling (CoLM), 2024.

Navigating the Dual Facets: A Comprehensive Evaluation of Sequential Memory Editing in Large Language Models

Zihao Lin, Mohammad Beigi, Hongxuan Li, Yufan Zhou, Yuxiang Zhang, Qifan Wang, Wenpeng Yin, Lifu Huang.

Annual Meeting of the Association for Computational Linguistics (ACL), 2024.

Customization Assistant for Text-to-Image Generation

Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu, Tong Sun.

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

An assistant which can generate creative images for specific user-input subject along with text explanation and elaboration in 2-5 seconds, without any fine-tuning.

TRINS: Towards Multimodal Language Models That Can Read

Ruiyi Zhang, Yanzhe Zhang, Jian Chen, Yufan Zhou, Jiuxiang Gu, Changyou Chen, Tong Sun.

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints

Jian Chen, Ruiyi Zhang, Yufan Zhou, Changyou Chen.

The Twelfth International Conference on Learning Representations (ICLR) 2024.

Enhanced Visual Instruction Tuning for Text-Rich Image Understanding

Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, Tong Sun.

Workshop on Instruction Tuning and Instruction Following at NeurIPS 2023.

Shifted Diffusion for Text-to-image Generation

Yufan Zhou, Bingchen Liu, Yizhe Zhu, Xiao Yang, Changyou Chen, Jinhui Xu.

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.

We propose a method termed Corgi, which can better generate image embeddings from text inside multimodal embedding space.
It benefits both standard and language-free text-to-image generation. And yes, I do have a Corgi.

LAFITE: Towards Language-Free Training for Text-to-Image Generation

Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, Tong Sun.

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.

Our proposed work, Lafite, is the first work which can successfully train text-to-image generation model with image-only dataset.

TiGAN: Text-Based Interactive Image Generation and Manipulation

Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu, Chris Tensmeyer, Tong Yu, Changyou Chen, Jinhui Xu, Tong Sun.

AAAI conference on Artificial Intelligence (AAAI), 2022.

Meta-Learning with Neural Tangent Kernels

Yufan Zhou*, Zhenyi Wang*, Jiayi Xian, Changyou Chen, Jinhui Xu.

International Conference on Learning Representations (ICLR), 2021.

MixKD: Towards Efficient Distillation of Large-scale Language Models

Kevin J Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, Lawrence Carin.

International Conference on Learning Representations (ICLR), 2021.

Learning Manifold Implicitly via Explicit Heat Kernel Learning

Yufan Zhou, Changyou Chen, Jinhui Xu.

Conference on Neural Information Processing Systems (NeurIPS), 2020.

Weakly-supervised Brain Tumor Classification with Global Diagnosis Label (Oral)

Yufan Zhou, Zheshuo Li, Chunwei Ma, Mingchen Gao, Changyou Chen, Hong Zhu, Jinhui Xu.

IEEE International Symposium on Biomedical Imaging (ISBI), 2020.

Variational Adversarial Kernel Learned Imitation Learning (Spotlight)

Fan Yang, Alina Vereshchaka, Yufan Zhou, Changyou Chen, Wen Dong.

AAAI conference on Artificial Intelligence (AAAI), 2020.

Learning Diverse Stochastic Action-Generators by Learning Smooth Latent Transitions (Spotlight)

Zhenyi Wang, Ping Yu, Yang Zhao, Ruiyi Zhang, Yufan Zhou, Junsong Yuan, Changyou Chen.

AAAI conference on Artificial Intelligence (AAAI), 2020.

Holistic Brain Tumor Screening and Classification Based on DenseNet and Recurrent Neural Network

Yufan Zhou, Zheshuo Li, Hong Zhu, Changyou Chen, Mingchen Gao, Kai Xu, Jinhui Xu.

International Conference on Medical Image Computing and Computer Assisted Intervention, Brain Lesion Workshop (BrainLes, MICCAI), 2018.

Manuscripts/Preprints

♠ Mentored Intern/Student

Toffee: Efficient Million-Scale Dataset Construction for Subject-Driven Text-to-Image Generation

Yufan Zhou, Ruiyi Zhang, Kaizhi Zheng, Nanxuan Zhao, Jiuxiang Gu, Zichao Wang, Xin Eric Wang, Tong Sun.

Efficient method to construct dataset for subject-driven T2I generation, which can save at least tens of thousands of GPU hours.

ARTIST: Improving Generation of Text-rich Images by Disentanglement

Jianyi Zhang^♠, Yufan Zhou, Jiuxiang Gu, Curtis Wigington, Tong Yu, Yiran Chen, Tong Sun, Ruiyi Zhang.

Improve text rendering ability of text-to-image diffusion model.

LLaVAR: Enhanced Visual Instruction Tuning for Text-rich Image Understanding

Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, Tong Sun.

Enhancing Detail Preservation for Customized Text-to-Image Generation: A Regularization-Free Approach

Yufan Zhou, Ruiyi Zhang, Tong Sun, Jinhui Xu.

A novel framework for customized text-to-image generation without the use of regularization.
We can efficiently customize a large-scale text-to-image generation model on single GPU, with only one image provided by the user.