Learnable inducing points and seed vectors

[rvinas]

No description provided.

[arrigonialberto86]

I see that in the previous implementation random.normal returned a Tensor, while changing this to Variable effectively makes this a trainable parameter

[rvinas]

Exactly. I believe both the seed vectors and inducing points are trainable in the paper, right?

Many thanks for the implementation!

[michaelpoluektov]

From the official PyTorch implementation:

class PMA(nn.Module):
    def __init__(self, dim, num_heads, num_seeds, ln=False):
        super(PMA, self).__init__()
        self.S = nn.Parameter(torch.Tensor(1, num_seeds, dim))
        nn.init.xavier_uniform_(self.S)
        self.mab = MAB(dim, dim, dim, num_heads, ln=ln)

    def forward(self, X):
        return self.mab(self.S.repeat(X.size(0), 1, 1), X)

They're using a trainable nn.Parameter.