[QUESTION] Does Megatron-Core supports LLAMA models?

[noob-ctrl]

Does Megatron-Core supports LLAMA models?

[ethanhe42]

yes

[noob-ctrl]

@ethanhe42 When transformer-impl is local, it reports the following error：
AssertionError: (RMSNorm) is not supported in FusedLayerNorm when instantiating FusedLayerNorm when instantiating TransformerLayer
When transformer-impl is transformer_engine, the following code does not seem to define RMSNorm?

So do I need to make any changes when I want to use llama?

[ethanhe42]

You need to use mcore models. local is deprecating

[noob-ctrl]

@ethanhe42 When transformer-impl is set to transformer_engine, the following code does not seem to define RMSNorm?

[ethanhe42]

It's handled by TEnorm