Background

Variational inference (VI) is a key framework in Bayesian deep learning, enabling scalable approximations of complex posterior distributions. Accurate entropy estimation is critical in VI but remains challenging, particularly for high-dimensional or multimodal distributions. Traditional methods, such as closed-form approximations or Monte Carlo sampling, can be computationally intensive or inaccurate. The spacings estimator, a non-parametric technique leveraging the ordering and spacing of samples, offers a promising alternative for efficient and robust entropy estimation.

This project explores the integration of the spacings estimator into VI workflows, focusing on its application to flexible variational distributions parameterized by deep neural networks. Additionally, we will investigate connections to hypernetworks, which generate parameters for other models, to enhance expressiveness in variational distributions. Comparative studies with normalizing flows will evaluate the strengths and limitations of the spacings estimator in improving the scalability and accuracy of Bayesian deep learning.

Objective(s)

  1. Develop and integrate the spacings estimator for entropy into variational inference workflows, focusing on its efficiency and robustness for flexible variational distributions.

  2. Investigate the role of hypernetworks in enhancing the expressiveness of variational distributions and their compatibility with the spacings estimator.

  3. Compare the performance of the spacings estimator against traditional methods, including normalizing flows, to evaluate its impact on scalability and accuracy in Bayesian deep learning.

Requirements

  1. Strong programming skills in Python and experience with deep learning frameworks such as PyTorch or TensorFlow.

  2. Background in probabilistic modeling and Bayesian methods, with familiarity in variational inference and entropy concepts.

  3. Basic understanding of non-parametric estimation techniques, or a willingness to learn these topics during the project.

Supervisors

Mikkel N. Schmidt, Associate Professor (CogSys), mnsc@dtu.dk

Potential outcomes

As this project contains elements of novel research, there is a possibility the project can be extended in an attempt to turn the thesis into a publication (either at a workshop, journal or conference).

References

  1. O. Vasicek, “A test for normality based on sample entropy”, Journal of the Royal Statistical Society Series B, vol. 38, pp. 54-59, 1976.

  2. E. G. Miller, “A new class of entropy estimators for multi-dimensional densities,” 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP ‘03)., Hong Kong, China, 2003, pp. III-297, doi: 10.1109/ICASSP.2003.1199463. Variational Inference with Normalizing Flows

  3. Danilo Rezende, Shakir Mohamed “Variational Inference with Normalizing Flows”, Proceedings of the 32nd International Conference on Machine Learning, PMLR 37:1530-1538, 2015.