Recent advancements in training large neural networks have produced promising results in both efficiency and performance. We present two novel approaches: a constrained feature distillation method for knowledge transfer and a memory-efficient training algorithm for large language models. Based on orthogonal projections and task-specific normalization, our distillation method outperforms the previous state-of-the-art on ImageNet and shows significant improvements in object detection and image generation.

For large language models, we introduce a technique that compresses intermediate activations without performance degradation, utilizing rank-1 sub-token projections during forward passes and coarse reconstruction in backward passes. This approach complements existing parameter-efficient fine-tuning methods and demonstrates competitive performance in both fine-tuning and pre-training scenarios. These innovations substantially improve model efficiency across various tasks and architectures, opening the way for more accessible and powerful neural network applications.

About the Speaker

Mr. Ismail Elezi lives and works in London. He serves as Senior Research Scientist (Tech Lead) & Research Scientist of Computer Vision/Machine Learning - Huawei.