Multi-Scale Residual Vector Quantization

[mahdip72]

Hi,

I am wondering if there is an easy way to use ResidualVQ and apply Multi-Scale to it via your amazing library?

1. Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
2. SAR3D: Autoregressive 3D Object Generation and Understanding via
   Multi-scale 3D VQVAE

These seem to be highly beneficial for Autoregressive modeling.

[lucidrains]

probably not on top of residual vq, but I could use it as a template for the multi scale vq in a separate file

you must be interested in the 3d case? it seems interesting although I have to think more about whether a shared codebook across hierarchies make sense

[mahdip72]

That would be a big favour!

you must be interested in the 3d case?

Honestly, I am looking to apply this on top of 2D feature maps of any encoders (not only 3D case).

As a reference, PointARU shows the same process better than the top ones:

And the main question is how to define the downsampling and upsampling functions/operations in it.

• PointARU uses a rather complicated way for down-sampling and up-sampling with learnable networks.
• Original VAR / SAR3D: simple resize/interpolation for downsampling; upsample + a small learnable conv.

I would say the original VAR approach is the most straightforward as a starting point. Also, VAR-style multi-scale VQ-VAE variants typically share a codebook across hierarchy levels as far as I am aware.

I find this insightful as well for the rationale behind it.

[lucidrains]

@mahdip72 ahh nice, yea, i can just add all of it up to 3 dimensions (since there are no conv4d yet)

thanks for sharing the blog post from the author, i see he does directly address the shared codebook, but i don't see the ablation in his paper.

nonetheless, yea i can add this to the repo, as i have been meaning to get back to residual vq (need to add the beam search feature that is showing up in some recent papers)