Dongyang Fan

I'm a PhD student at Machine Learning and Optimization Lab at EPFL, supervised by Prof. Martin Jaggi.

My research interests are:

• Efficient and Collaborative Machine Learning: improving data and resource efficiency via collaborative learning.
• Incentives and Fairness: exploring the interplay between learning and incentives in multi-agent systems.
• Post-training of LLMs: fine-tuning for personalization, knowledge editing and alignment.

Email  /  CV  /  Google Scholar  /  Twitter  /  Github  /  LinkedIn

Research

We propose a novel collaborative language modeling framework that leverages the modularity of MoE. Our framework can efficiently tackle data and resource heterogeneity across devices.

We ablate the design choices of Mixture-of-Experts models, including the different expert specialization from sequence- and token-level routing, and the impact of the number of actiavted and total experts.

We introduce three distinct trust-weighted gradient aggregation schemes for collaborative on-device personalized language modeling.

We propose a novel regularizer, ghost noise, that can be used to improve the generalization of DNNs and can be applied to noise-free layer-normalized networks.

We propose a novel co-distillation method, where models learn from each other by reaching a consensus on the predictions.

Source codes of the website are from here.