HyperNet Fields: Efficiently Training Hypernetworks without Ground Truth by Learning Weight Trajectories

Submission #322

Supplementary PDF

Descriptive Text

We propose hypernetwork fields, where we learn the entire weight space trajectory instead of only the final state, which allows us to train without ever needing to know the final converged weights. Our method can be applied to any application of hypernetworks, including diffusion model personalization and modeling of 3D neural representations

Abstract

To efficiently adapt large models or to train generative models of neural representations, Hypernetworks have drawn interest. While hypernetworks work well, training them is cumbersome, and often requires ground truth optimized weights for each sample. However, obtaining each of these weights is a training problem of its own---one needs to train, e.g., adaptation weights or even an entire neural field for hypernetworks to regress to. In this work, we propose a method to train hypernetworks, without the need for any per-sample ground truth. Our key idea is to learn a Hypernetwork `Field' and estimate the entire trajectory of network weight training instead of simply its converged state. In other words, we introduce an additional input to the Hypernetwork, the convergence state, which then makes it act as a neural field that models the entire convergence pathway of a task network. A critical benefit in doing so is that the gradient of the estimated weights at any convergence state must then match the gradients of the original task---this constraint alone is sufficient to train the Hypernetwork Field. We demonstrate the effectiveness of our method through the task of personalized image generation and 3D shape reconstruction from images and point clouds, demonstrating competitive results without any per-sample ground truth.

Overview

Descriptive Text

In order to train our hypernetwork field we input the condition as well as a timestep t sampled from U(0, T). We then supervise the model to match the gradient of the task specific network with the estimated numerical derivative of he hypernetwork.

Descriptive Text

The sampling process to generate personalized images with our hypernetwork framework can be done by simply querying our hypernetwork field with timestep T.

Visualizing learned trajectory

Loading...
Initialization Convergence

This interactive slider provides a visualization of intermediate values of t from our fully trained Hypernetwork Field. The top row displays the condition images, while the bottom row shows the corresponding generated images based on the estimated DreamBooth for each value of t. Notably, visualized are DreamBooths output directly from our hypernetwork in a single forward pass before for fast fine-tuning. As t increases, the generated images improve progressively, illustrating that the Hypernetwork Field effectively learns the correct trajectory of the task network.

More qualitative examples

We show examples of our generated videos sampled from HyperNet. Each shows different variations of generated content.

Sample Set 1
Sample Set 2
Sample Set 3
Sample Set 4
Sample Set 5
Sample Set 6

ShapeNet meshes

GIF
GIF

Interpolating t – As with the DreamBooth setup we show what sampling intermediate values of t looks like, both with an example shape and the IOU curve. It can be see that the IOU curve can be seen to be monotonically improving as t increases, as you would expect from a well trained model.

GIF

Random walk through VAE latent space – Since our Hypernetwork field trained on ShapeNet was trained with a VAE, we can also interpolate in the latent space of the VAE to generate new shapes. It can be seen that the Hypernetwork field generates smoothly varying shapes as we interpolate in the latent space.