Hackathon Workshop on Generative Modelling

On July 8-12 2024, the School hosted a hackathon which provided a platform for early career researchers and PhD students to work on challenges with a strong emphasis on generative modelling. Two of the industry-inspired challenges were provided by Amazon.

The event was part of the Isaac Newton Institute (INI) satellite programme on 'Diffusions in machine learning: Foundations, generative models and non-convex optimisation'. 

On the first day of the event, participants attended introductory talks on the challenges and a training session about the use of High Performance Computing Cirrus before being allocated into groups to work on one of the four challenges. The groups updated the participants of the INI programme throughout the week, and presented their final results in a session which was livestreamed at the Alan Turing Institute.

The first challenge, 'Hallucinations in Large Vision-Language Models', aimed to improve the reasoning capabilities of image + text multimodal models (e.g. GPT-4V) and reduce both visual hallucinations (generated images that fail to adhere to the input prompt) and text hallucinations (incorrect answers to Visual Question Answering prompts).

'Unlearning for Large Language Models', the second challenge, explored methods for targeted and efficient unlearning focusing on two benchmarks: TOFU and Unlearning_LLM. 

The goal of the third challenge, 'Towards frugal zero-shot diffusion-based image restoration' was to guide participants to develop robust and frugal algorithms that leverage advances in diffusion models to solve inverse problems with no additional training of the models. The focus of this challenge was on common tasks encountered in image inpainting, deblurring, and colourisation.

Challenge four, 'Implementing non-convex optimisation algorithms', provided a platform for participants to engage in the hands-on implementation of non-convex optimisation algorithms and to assess their performance across a variety of datasets and deep learning models. It was specifically tailored for participants who are beginners with deep learning frameworks like PyTorch but may possess a fundamental knowledge of Python.

Thanks to the Centre for Investing Innovation, CRC Press, the London Mathematical Society, and Springer who provided support for the workshop.

The winning teams

Concrete outcomes

Since winning the hackathon with the self-consistent test for diffusion models, Olga Loginova, Anil Batra, and Gautier Dagan had a paper accepted at the prestigious conference, COLING 2025:

CAST: Cross-modal Alignment Similarity Test for Vision Language Models | arXiv 

A fourth team, Alex Richardson, Kevin Zhang, and Lucas Beerens, that worked on attacks for unlearning in diffusion models along with Dolly Chen (who came in second), have submitted a paper to ICML:

 Rethinking the Vulnerability of Concept Erasure and a New Method | arXiv