DPOD: Optimizing Pareto Frontier via Progressive Shrinking
Liu, Junchen
Liu, Junchen
Author
Supervisor
Department
Machine Learning
Embargo End Date
2025-12-31
Type
Thesis
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Deep learning with neural networks has gained popularity these years. However, feats like considerable memory and computation occupation restrict them from being deployed on edge devices. As a result, enabling the deployment of models on resource-constrained devices and improving the efficiency during the model inference process seem inevitable to achieve efficient inference goals. While powerful structures and techniques have been proposed, including Slimmable Networks, earlyexit and skip blocks during inference, and ordered dropout, they often incur undesired dependencies during the training process. Alternatively, we take a step further in pruning the width of neural networks dynamically during training and introduce DPOD, a training framework that finds the optimal OD ratio profile under efficiency constraints. Instead of applying a universal OD ratio to all layers or sampling the OD ratio independently during training, we prune a single layer at one step to avoid nested dependencies. At the same time, it allows each layer to explore the foot-print-accuracy trade-off. We state that DPOP is a novel solution that comprises two key features. The first is eliminating undesired dependencies by single-layer sampling. The second feature is allowing picking OD ratios at layer granularity. When applying DNNs, we proved that our solution efficiently uncovered the Pareto models under footprint constraints. What’s more, these Pareto models have competitive performance after finetuning.
Citation
Junchen Liu, “DPOD: Optimizing Pareto Frontier via Progressive Shrinking,” Master of Science thesis, Machine Learning, MBZUAI, 2025.
Source
Conference
Keywords
Efficient Machine Learning, Structured Pruning, Ordered Representation