Supplementary material:

- cvpr20a-supp.pdf: details of experiments, metrics, and additional figures
  and tables.

- Animations showing how the rank at each layer changes over iterations of our
  LC algorithm. We show results for different cost (penalty) functions: 
  . Penalizing FLOPs (with different penalty hyperparameter lambda):
    VGG16-flops1.gif, VGG16-flops2.gif.
  . Penalizing storage: VGG16-storage.gif.
  The exact details of the neural network (VGG16) and the training are given
  in cvpr20a-supp.pdf.
  Each animation has one frame per iteration, where both the ranks and the
  network weights change, and a final, fine-tuning iteration, where the
  weights change but the ranks are fixed. At each iteration, we show:
  . The penalty hyperparameter lambda (constant over iterations), which sets
    the strength of the cost function (FLOPs or storage).
  . As an inset plot, the classification test error (blue curve) and the
    compression ratios for FLOPs and storage (red curves). We show both FLOPs
    and storage for reference; in each animation we penalize only one of them. 
    The current iteration is marked by the vertical black line.
  . In the main plot, we show the rank for each of the 15 layers in VGG. The
    thick blue lines indicate the maximum rank for each layer (given by the
    original VGG network) and the gray bars the current rank.
  Note how the distribution of the resulting ranks (and the corresponding
  network architecture) varies significantly depending on the type of cost
  function and the value of lambda. Larger lambda leads to lower ranks
  overall; penalizing storage puts more pressure on reducing the ranks of
  layers contributing many free parameters (the fully-connected and final
  convolutional layers, layers 8-14); penalizing FLOPs puts more pressure on
  reducing the ranks of layers consuming more computation (the initial
  convolutional layers, layers 1-7).

All these animations may be seen with a web browser or with specialized
GIF image viewers.

