400GB for llama 3?!?! should we be making our own gpu now?

[ouvaa]

can someone take a guess what's the most powerful amd epyc 9000 series and motherboard and ddr ram types and what kind of t/s we can expect to get to load a 400GB llama 3 in future?

[phymbert]

Meta casually trains it on 16k GPUs, so we have a limit :)

[ouvaa]

let's say they put out a 1TB in future, what do u expect non multimillionaires to do about that? live with 0.1 t/s?

that's why i created this language. i think it's important normal people have access to AI usage at scale
https://github.com/ouvaa/ouvaa.github.io

[mirek190]

can someone take a guess what's the most powerful amd epyc 9000 series and motherboard and ddr ram types and what kind of t/s we can expect to get to load a 400GB llama 3 in future?

So even if you use 4b version you still need 512 GB RAM ...

[supportend]

I use the instruct models (8b and 70b), both are working with main on my system. I'm not sure, i use the correct prompt format, but it works.

[mirek190]

I am testing from main.exe of llamacpp

main.exe --model models/new3/Meta-Llama-3-8B-Instruct.Q8_0.gguf --color --threads 30 --keep -1 --n-predict -1 --repeat-penalty 1.1 --ctx-size 0 --interactive -ins -ngl 99 --simple-io -r '<|eot_id|>' --in-prefix "\n<|start_header_id|>user<|end_header_id|>\n\n" --in-suffix "<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n" -p "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful, smart, kind, and efficient AI assistant. You always fulfill the user's requests to the best of your ability.<|eot_id|>\n\n"

[anianios]

Has anyone been able to make it work with server process instead of the main?

[ouvaa]

@mirek190 i've spent a lot of time playing with llama before but seriously, i dont think it's worth touching anything that's 22b and below... but since u say so, i'll try someday. i'm more looking forward to 281GB Mixtral now. is the llm ranking for both llama 3 and mixtral 281 out yet?

[mirek190]

From leaderboard lmsys llama 3 8b has level of mistal 8x22b what is insane.