Portrait
Xiyue Zhu
Ph.D. Student, Computer Engineering
University of Illinois Urbana-Champaign
About Me

Hi! I'm a final-year Ph.D. candidate in Computer Engineering at the University of Illinois Urbana-Champaign (UIUC), advised by Prof. Volodymyr Kindratenko. Before that, I received my M.S. from UIUC, advised by Prof. Shenlong Wang, where I worked on generative models for autonomous driving, and my bachelor's degrees from the ZJU-UIUC Institute.

My research centers on large language and vision-language models, in particular reinforcement learning–based post-training and AI agents. I also have substantial experience with visual generative models such as diffusion and video generation. Across these areas, I am especially interested in turning imperfect feedback, such as online user signals, LLM judgments, and visual quality metrics, into reliable training rewards for open-ended tasks. My work spans applications in e-commerce search, web automation, medical report generation, and video creation, as well as generative models for autonomous driving earlier in my PhD. I regularly serve as a reviewer for CVPR, ECCV, and NeurIPS.

I am graduating in December 2026 and looking for full-time positions in VLMs, AI agents, and video generation. Feel free to reach out!

Resume
Education
  • University of Illinois Urbana-Champaign
    Ph.D. in Computer Engineering
    Aug. 2021 - Dec. 2026 (expected)
  • University of Illinois Urbana-Champaign
    M.S. in Computer Engineering
    Aug. 2021 - May 2023
  • Zhejiang University & UIUC (ZJU-UIUC Institute)
    B.Eng. (ZJU) & B.Sc. (UIUC) in Computer Engineering, dual-degree program
    Aug. 2017 - May 2021
Experience
  • TikTok
    Machine Learning Engineer Intern
    May 2026 - Aug. 2026
  • Xpeng Motors
    Machine Learning Intern
    May 2025 - Jan. 2026
  • Amazon
    Applied Scientist Intern
    May 2024 - Aug. 2024
Industry
Query recommendation and agentic search for e-commerce product pages and videos
Query recommendation and agentic search for e-commerce product pages and videos

TikTok · Machine Learning Engineer Intern · May – Aug 2026

LLM / VLM AI agent RL post-training Reward design Distributed training E-commerce Query recommendation

Schematic of the serving and training pipeline. Internal work; no public paper.

  • Post-trained Qwen3-VL-8B for query recommendation: distillation from a frontier closed-source model on 200K samples, then GRPO with rewards from the online CTR of deployed questions and an agent-as-judge.
  • A downstream web agent answers the proposed questions by tool-calling internal search.
  • Trained with verl. Doubled offline-estimated CTR over the production system.
TransDrive: a shared-BEV vision–language–action framework unifying planning and end-to-end driving
TransDrive: a shared-BEV vision–language–action framework unifying planning and end-to-end driving

Xpeng Motors · Machine Learning Intern · May 2025 – Jan 2026

LLM / VLM Autonomous driving Vision-language-action BEV representation End-to-end driving Distributed training

Two-stage training: pretrain on rendered nuPlan planning data, then fine-tune on nuScenes with BEV maps from a frozen perception model.

  • Instead of feeding a VLM latent BEV embeddings, render perception outputs and vectorized planning data into the same human-readable BEV image, so one Qwen2.5-VL backbone serves both tasks without architectural changes.
  • The shared representation unlocks pretraining on planning datasets that are 10× larger than end-to-end driving data, then transfer to sensor-based driving — with the largest gains in low-data regimes.
  • Led distributed training on 200+ GPUs; reduced L2 trajectory error by 4–9% over the baseline model.
Turbocharging Web Automation: The Impact of Compressed History States
Turbocharging Web Automation: The Impact of Compressed History States ACL 2025 Findings

Amazon · Applied Scientist Intern · May – Aug 2024

Xiyue Zhu, Peng Tang, Haofu Liao, Srikar Appalaraju

LLM / VLM AI agent Web agents Context compression

A shared history compressor distills each verbose past web state into a fixed-length representation before the action-prediction transformer.

  • Designed a compressed history-state representation for VLM web agents, cutting context length by ~70% while preserving the action history needed for multi-step navigation.
  • Improved step accuracy by 2–5% on Mind2Web and WebLINX.
Research
Agentic reasoning for video generation
Agentic reasoning for video generation

UIUC · Dissertation chapter · Jan 2025 – Jun 2026

Video generation / understanding AI agent LLM / VLM Diffusion models / visual generative models RL post-training Reward design Chain-of-thought planning

The generator and its metrics are treated as an environment; language agents plan, inspect, regenerate, and roll back around a frozen video diffusion model.

  • A video-creation agent that integrates chain-of-thought planning and iterative self-refinement into video generation, built on Causal Forcing as the generator and Qwen2.5-VL-7B as the reasoner.
  • Four roles around a frozen generator: a prompt agent, a video-understanding agent with adaptive frame inspection, a keyframe agent for hard temporal transitions, and a supervisor that can roll back to a better earlier candidate.
  • Trained the reasoner with GRPO using VisionReward on the final video (minus a diffusion-call penalty) as the reward, improving VisionReward by 20.3% over the base generator with better physical plausibility and causal consistency.
  • Supports multi-modal conditioned generation and interactive, interpretable editing through explicit reasoning traces.
MedQPA-Gen: Medical Question Proposing and Answering for Report Generation
MedQPA-Gen: Medical Question Proposing and Answering for Report Generation ACL 2026 Findings

UIUC · Jun 2025 – Jan 2026

Weijie Liang*, Xiyue Zhu*#, Ruike Zhu, Chenhao Li, Cheng Tang, Zhiyu Liu, Zhihua Gong, Shirui Luo, Yudu Li, Volodymyr Kindratenko (* equal contribution, # corresponding author)

AI for healthcare LLM / VLM AI agent RL post-training Reward design Report generation LLM-as-judge

Figure 2 of the paper: the MedQPA evaluation loop (question proposing and answering), reflective prompting, and the RL update against the MedQPA reward model.

  • Proposed MedQPA, a question-driven agentic evaluation pipeline for medical report generation: propose clinical questions, answer them against image and report, and score fact consistency.
  • Developed MedQPA-Gen, which optimizes report generators against MedQPA via reflective prompting, iterative DPO, and GRPO on Qwen2-VL-7B.
  • 80% human-preference win rate over the base model and +3.5% accuracy on downstream disease classification.
Introducing 3D Representation for Volume-to-Volume Translation via Score Fusion
Introducing 3D Representation for Volume-to-Volume Translation via Score Fusion ICML 2025

UIUC · Nov 2023 – Feb 2025

Xiyue Zhu, Dou Hoon Kwark, Ruike Zhu, Kaiwen Hong, Yiqi Tao, Shirui Luo, Yudu Li, Zhi-Pei Liang, Volodymyr Kindratenko

Diffusion models / visual generative models AI for healthcare Video generation / understanding MRI super-resolution Inverse problems 3D representation

At each denoising step, perpendicularly trained 2D diffusion models give initial estimates and a lightweight 3D network learns to fuse them in score space; alignment modules inject hierarchical 2D features.

  • Score Fusion learns to fuse diffusion score functions instead of combining them arithmetically, yielding a 3D representation for volume-to-volume translation and multi-modal fusion — the first work to bring 3D generative priors into MRI super-resolution and outperform 2D methods.
  • Improved 3D medical-image and video inverse-problem results by 3–10%, with better downstream tumor segmentation.
MapPrior: Bird's-Eye View Map Layout Estimation with Generative Models
MapPrior: Bird's-Eye View Map Layout Estimation with Generative Models ICCV 2023

UIUC · Jul 2022 – Aug 2023

Xiyue Zhu, Vlas Zyrianov, Zhijian Liu, Shenlong Wang

Autonomous driving Diffusion models / visual generative models BEV map segmentation Generative priors

An off-the-shelf perception model gives a noisy estimate; a generative encoder, transformer sampler, and decoder turn it into realistic, diverse map layouts.

  • Introduced generative priors for BEV map segmentation, exploiting the prior distribution of map layouts to fix occluded and out-of-range regions.
  • Improved IoU by ~1% and substantially improved realism (MMD) and uncertainty estimation (ECE) for downstream planning on nuScenes.
LiDARGen: Learning to Generate Realistic LiDAR Point Clouds
LiDARGen: Learning to Generate Realistic LiDAR Point Clouds ECCV 2022

UIUC · Jan – Oct 2022

Vlas Zyrianov, Xiyue Zhu, Shenlong Wang

Autonomous driving Diffusion models / visual generative models LiDAR generation Score-based models

Point clouds are generated by score-based denoising in the equirectangular range/intensity view.

  • Implemented BEV-FID metrics and LiDAR generative baselines (ShapeGF, PU-GAN) for evaluating the realism of generated LiDAR scans on KITTI-360 and nuScenes.