Does the number of slots * the number of experts have to be the same as the length of the token?

[ZQpengyu]

Assuming I have a sequence of length 512, but num_slot * num_expert = 9, it should still be able to run. In this case, would there be a performance drop?

[andersonbcdefg]

Have to validate that empirically, but I would definitely expect a performance drop here. Because of softmax, my intuition is it's likely that 1 token dominates each expert, and some tokens get short shrift. This might be fine if you have like, 2x more slots, or maybe even 4x or 8x, but >50 tokens per slot I imagine would be skimping a bit and likely to hurt results.