Bayesian Optimization and Scaling laws for Hyperparameter Fine-tuning and exploration of new PEFT methods
Zhai, Hanshuo
Zhai, Hanshuo
Author
Supervisor
Department
Machine Learning
Embargo End Date
2025-10-01
Type
Thesis
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Bayesian optimization is an efficient global optimization technique often used for optimizing black-box functions, especially when evaluating the function is costly—such as in hyperparameter tuning for machine learning models. The main idea behind Bayesian optimization is to build a surrogate model (typically a Gaussian Process or a tree-based model) that approximates the objective function. This model is then used to select the next evaluation point based on the acquisition function, enabling efficient optimization with elatively few function evaluations.
Among these acquisition functions, EEIPU is a very important. EEIPU (Expected Expected Improvement Per Unit-cost) is a novel approach in Bayesian optimization designed for cost-sensitive hyperparameter tuning, particularly in multi-stage machine learning pipelines. In complex machine learning workflows, where multiple stages (such as data preprocessing, feature extraction, and modeling) require different amounts of computational resources, optimizing each stage’s hyperparameters efficiently within a limited budget is challenging. EEIPU addresses this by focusing on maximizing performance gains per unit cost.
Citation
Zhai, Hanshuo, “Bayesian Optimization and Scaling Laws for Hyperparameter Fine-Tuning and Exploration of New PEFT Methods,” Master of Science thesis, Machine Learning, MBZUAI, 2025.
Source
Conference
Keywords
Bayesian Optimization, Energy-Efficient Inference, Parameter-Efficient Fine-Tuning, Cost-Aware Learning