Yuxuan Zhou

Yuxuan Zhou

LLM Pretraining · Multi-Token Prediction and Looped Architectures

✉ zhouyuxuanyx@gmail.com ★ GitHub 📚 Google Scholar Google Scholar citations

About

I currently work on the pretraining of large language models. My work spans Multi-Token Prediction and Looped Architectures, as well as probabilistic modeling, e.g., scoring functions beyond softmax and speculative decoding, with first-authored papers at ICML, NeurIPS (Oral), CVPR, ACL and ICLR (Oral).

I was trained as an automotive engineer. I took part in the pre-development of the steering control system for the Audi A8 (2018 Edition), on the modeling and simulation of driving dynamics.

Languages: Mandarin Chinese (native) · English (fluent) · German (C1)

News

Adopted at Scale

ECHO — Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction (arXiv · code)
A causal self-recursive Multi-Token Prediction (MTP) paradigm:
  • Final-layer sharing: turns the indirect supervision of auxiliary MTP into direct multi-horizon supervision of the backbone, resulting in more substantial gains at a single prediction depth and continued gains with added depth, where auxiliary MTP saturates; the gains concentrate on reasoning and code in pretraining, with QA & multiple choice holding, and extend to both categories after supervised finetuning.
  • Dynamic Depth Sampling: makes deep supervision affordable while preserving its gains.
  • Emergent depth extrapolation: maintains high acceptance rates even when drafting beyond the trained depth, entirely eliminating the need for post-hoc drafters.
Adopted in pretraining the 200B-A13B production model.
MultiMax — Sparse and Multi-Modal Attention Learning (ICML 2024 · arXiv · code)
A learnable generalization of SoftMax for attention and output distributions:
  • Sparse yet diverse: a unified SoftMax temperature can only trade off the sparsity and multi-modality of the generated distribution; MultiMax’s piece-wise temperature breaks this trade-off and pushes out the Pareto frontier, suppressing the long tail of implausible tokens while preserving the inherent diversity of plausible next tokens.
  • Negligible cost: MultiMax modulates the logits element-wise inside the chunked, Triton-fused cross-entropy kernel, just before its SoftMax, costing merely around 1% training slowdown and no extra memory for a pretraining-loss gain above 0.01.
Adopted on the vocabulary head in pretraining the 200B-A13B production model.

Experience

ERNIE Team, Baidu — Staff Research Engineer Apr 2026 – Present
Multi-Token Prediction and looped architectures.
Dexmal — Research Intern Oct 2025 – Jan 2026
Self-evolving diffusion policy for long-horizon robot manipulation (SeedPolicy).
Qwen Team, Alibaba — Research Intern Jan 2025 – Apr 2025
Lossless verification for speculative decoding (HSD, ICLR 2026 Oral).
Language Technologies Institute, CMU — Visiting Researcher Jun 2023 – Oct 2023
Topology-aware graph networks for action recognition (BlockGCN, CVPR 2024).
Tongyi Lab, Alibaba — Research Intern Jul 2022 – Jul 2023
Spatial priors for vision transformers (SP-ViT, BMVC 2022).
Bosch Center for Artificial Intelligence — Research Intern Jan 2019 – May 2019
Unsupervised anomaly detection for deep generative models, applied to optical inspection on assembly lines (NeurIPS 2020).
Chassis Electronics, Audi AG — Research Intern Apr 2017 – Oct 2017
Dynamics modeling and simulation of the steering control system for the Audi A8 (2018 Edition), including a non-Newtonian fitting model of the tire's nonlinear force–slip characteristics.

Education

Ph.D. in Computer Science — University of Mannheim 2020 – 2025
Advised by Margret Keuper and Mario Fritz. Research focus: Transformers, Multi-Token Prediction and Speculative Decoding.
M.Sc. in Automotive Engineering — University of Stuttgart 2015 – 2018
Research focus: physical modeling of driving dynamics and internal combustion engines.
B.Sc. in Automotive Engineering — Tongji University 2010 – 2015
Foundations of automotive engineering; a mandatory extra year of German language study.

Selected Publications

Views: – today · – total
© Yuxuan Zhou 2026 · Last updated Oct 2026