Pre-training Qwen for better domain specific knowledge? #1211
Unanswered
Tejaswgupta
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
We saw lawyer-llama built by using SFT on a dataset of legal QA pairs. Recently Saul-LM which was fine-tuned on Mistral achieves good performance as well.
However, I've seen Qwen-72B performing exceptionally well just with prompt tuning, would it be feasible to do continual pre-training and would it possibly improve accuracy and conversation quality?
Use cases:
Beta Was this translation helpful? Give feedback.
All reactions