Ch6

Applications — Learning Predictive, Generative and Representation Models

Unity of knowledge and action!

In this chapter, we present applications of stochastic compositional optimization and finite-sum coupled compositional optimization (FCCO) in both supervised and self-supervised learning settings. These include training predictive models, generative models, and representation models based on advanced objective functions such as distributionally robust optimization (DRO), group DRO (GDRO), AUC losses, NDCG loss, and contrastive losses. We also highlight applications of compositional optimization in solving multiple inequality-constrained optimization problems, optimizing data compositional neural networks, and a new paradigm of learning with a reference model called DRRHO risk minimization.

---

Contents

• 6.1 Stochastic Optimization Framework
• 6.2 DRO and Group DRO
• 6.3 Extreme Multi-class Classification
• 6.4 Stochastic AUC and NDCG Maximization
• 6.5 Discriminative Pretraining of Representation Models
• 6.6 Discriminative Fine-tuning of Large Language Models
• 6.7 Constrained Learning
• 6.8 Learning Data Compositional Networks
• 6.9 DRRHO Risk Minimization
• 6.10 History and Notes