Apply Tanh activation function to ViT - MLP Head

[joeycouse]

In the paper:

"In order to stay as close as possible to the original Transformer model, we made use of an additional
[class] token, which is taken as image representation. The output of this token is then transformed into a class prediction via a small multi-layer perceptron (MLP) with tanh as non-linearity
in the single hidden layer."

vit-pytorch/vit_pytorch/vit.py

Lines 110 to 113 in 5699ed7

	self.mlp_head = nn.Sequential(
	nn.LayerNorm(dim),
	nn.Linear(dim, num_classes)
	)

Should there be a Tanh() function applied after the linear layer?