Hyperparameter Scaling for Neural Network Optimisation

Overview of the Research:

Training modern neural networks requires choosing hyperparameters such as the learning rate, weight decay and momentum. Unfortunately, the best values of these hyperparameters often change as we increase the size of the model or the amount of training data. This makes scaling expensive: hyperparameters tuned on small models may need to be tuned again from scratch for much larger models.

Existing approaches try to solve this problem by deriving mathematical scaling rules, often by analysing neural networks in infinite-width or infinite-depth limits. These methods have produced important results such as the maximal-update parametrisation (µP), but the required theory can be technically demanding, architecture-specific, and may not capture finite-scale or time-dependent effects.

In this project, we will investigate a different approach: rather than theoretically predicting how important quantities change with scale, we will measure them directly during training. Preliminary work has shown that measuring the change a parameter update causes to the neural network’s function can be used to transfer learning rates between models of different widths, without needing to hand-derive scaling rules or make restrictive assumptions about the model or optimiser.

We will develop this idea into a general framework for scaling neural network optimisation, covering changes in model width and depth, dataset size, batch size and training duration, and hyperparameters including learning rate, weight decay and momentum. This requires a mixture of deep learning, optimisation, theory, and mathematics (primarily linear algebra and probability).

Join us if you are interested in understanding how large neural networks learn, and in developing practical methods that could make training large AI systems substantially more efficient.

Project keywords: deep learning; machine learning; artificial intelligence; neural networks; optimisation; data science; statistics; computer science

Candidate Requirements:

Applicants should hold, or expect to receive, a First Class or good Upper Second Class UK Honours degree (or the equivalent) in a relevant subject. A master’s level qualification would also be advantageous.

Non-UK applicants must meet the programme’s English language requirement by the application deadline.

Enquiries and Applications:

Informal enquiries are encouraged and should be directed to Dr Edward Milsom, em846@bath.ac.uk.

Formal applications should be submitted via the University of Bath’s online application form for a PhD in Computer Science prior to the closing date of this advert.

IMPORTANT:

When completing the application form:

1. In the Funding your studies section, select ‘University of Bath URSA’ as the studentship for which you are applying.
2. In the Your PhD project section, quote the project title of this project and the name of the lead supervisor in the appropriate boxes.

Failure to complete these two steps will cause delays in processing your application and may cause you to miss the deadline.

More information about applying for a PhD at Bath may be found on our website.

PLEASE BE AWARE: Applications for this project may close earlier than the advertised deadline if a suitable candidate is found. We therefore recommend that you contact the lead supervisor prior to applying and submit your formal application as early as possible.

Funding Notes

Candidates may be considered for a University of Bath studentship tenable for 3.5 years. Funding covers tuition fees, a stipend (£21,805 p/a in 2026/7) and access to a training support budget.

Studentships are open to both Home and exceptional International students; however, International applicants should note that funding does NOT cover the cost of moving to the UK, obtaining a student visa (View Website) or payment of the UK healthcare surcharge (View Website).