Small Language Models (SLMs) are a type of LLM family with fewer number of parameters. SLMs are so lightweight that they can be easily used with consumer-grade GPUs (e.g., NVIDIA RTX GPUs) and embedded systems (e.g., NVIDIA Jetson Orin). Recently, many open-source SLMs such as phi2, Gemma, and TinyLlama have been released. Training SLMs does not require as much computational resources as larger models. This motivates us to consider training our own SLM for specific purposes.
This tutorial series covers how to train SLMs with various techniques using NeMo Framework.
- Building Your Own Small Language Model from Scratch notebook