You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This test used only 64 vcpus or 32 vcpus per VM due to Distributed Llama limitations.
Average Single Token Generation Time
Llama 7B Q40 Weights Q80 Buffer
Metric
1 VM 64 vCPUs
2 VMs 64 vCPUs
1 VM 32 vCPUs
2 VMs 32 vCPUs
Avg generation time
37.12ms
33.62 ms 🔥
41.56 ms
34.94 ms
Avg inference time
34.31 ms
24.00 ms
39.81 ms
27.12 ms
Avg transfer time
1.56 ms
8.56 ms
1.12 ms
6.88 ms
1 VM
64 cpus
n4no_com@instance-1:~/distributed-llama$ sudo nice -n -20 ./main inference --model ../dllama_llama-2-7b_q40.bin --tokenizer ../tokenizer.bin --weights-float-type q40 --buffer-float-type q80 --prompt "Hello world" --steps 16 --nthreads 64
💡 dim: 4096
💡 hiddenDim: 11008
💡 nLayers: 32
💡 nHeads: 32
💡 nKvHeads: 32
💡 vocabSize: 32000
💡 seqLen: 2048
💡 nSlices: 1
⏩ Loaded 4242882560 bytes
🔶 G 36 ms I 35 ms T 0 ms S 0 kB R 0 kB Hello
🔶 G 36 ms I 34 ms T 1 ms S 0 kB R 0 kB world
🔶 G 36 ms I 31 ms T 4 ms S 0 kB R 0 kB .
🔶 G 39 ms I 34 ms T 2 ms S 0 kB R 0 kB nobody
🔶 G 36 ms I 34 ms T 1 ms S 0 kB R 0 kB
🔶 G 37 ms I 35 ms T 1 ms S 0 kB R 0 kB Do
🔶 G 37 ms I 33 ms T 3 ms S 0 kB R 0 kB you
🔶 G 37 ms I 35 ms T 1 ms S 0 kB R 0 kB think
🔶 G 37 ms I 35 ms T 1 ms S 0 kB R 0 kB the
🔶 G 38 ms I 34 ms T 2 ms S 0 kB R 0 kB
🔶 G 36 ms I 34 ms T 1 ms S 0 kB R 0 kB 2
🔶 G 38 ms I 36 ms T 0 ms S 0 kB R 0 kB 0
🔶 G 38 ms I 37 ms T 0 ms S 0 kB R 0 kB 1
🔶 G 37 ms I 32 ms T 4 ms S 0 kB R 0 kB 9
🔶 G 38 ms I 36 ms T 1 ms S 0 kB R 0 kB legisl
🔶 G 38 ms I 34 ms T 3 ms S 0 kB R 0 kB ation
Generated tokens: 16
Avg generation time: 37.12 ms
Avg inference time: 34.31 ms
Avg transfer time: 1.56 ms
32 cpus
n4no_com@instance-1:~/distributed-llama$ sudo nice -n -20 ./main inference --model ../dllama_llama-2-7b_q40.bin --tokenizer ../tokenizer.bin --weights-float-type q40 --buffer-float-type q80 --prompt "Hello world" --steps 16 --nthreads 32
💡 dim: 4096
💡 hiddenDim: 11008
💡 nLayers: 32
💡 nHeads: 32
💡 nKvHeads: 32
💡 vocabSize: 32000
💡 seqLen: 2048
💡 nSlices: 1
⏩ Loaded 4242882560 bytes
🔶 G 41 ms I 40 ms T 0 ms S 0 kB R 0 kB Hello
🔶 G 40 ms I 39 ms T 1 ms S 0 kB R 0 kB world
🔶 G 41 ms I 39 ms T 1 ms S 0 kB R 0 kB !
🔶 G 41 ms I 39 ms T 1 ms S 0 kB R 0 kB It
🔶 G 41 ms I 40 ms T 0 ms S 0 kB R 0 kB ’
🔶 G 41 ms I 37 ms T 3 ms S 0 kB R 0 kB s
🔶 G 44 ms I 44 ms T 0 ms S 0 kB R 0 kB been
🔶 G 42 ms I 40 ms T 1 ms S 0 kB R 0 kB a
🔶 G 41 ms I 37 ms T 3 ms S 0 kB R 0 kB long
🔶 G 42 ms I 40 ms T 1 ms S 0 kB R 0 kB time
🔶 G 41 ms I 40 ms T 1 ms S 0 kB R 0 kB since
🔶 G 42 ms I 40 ms T 2 ms S 0 kB R 0 kB I
🔶 G 42 ms I 40 ms T 1 ms S 0 kB R 0 kB ’
🔶 G 42 ms I 40 ms T 1 ms S 0 kB R 0 kB ve
🔶 G 42 ms I 41 ms T 1 ms S 0 kB R 0 kB posted
🔶 G 42 ms I 41 ms T 1 ms S 0 kB R 0 kB ,
Generated tokens: 16
Avg generation time: 41.56 ms
Avg inference time: 39.81 ms
Avg transfer time: 1.12 ms
2 VMs
64 cpus
n4no_com@instance-1:~/distributed-llama$ sudo nice -n -20 ./main inference --model ../dllama_llama-2-7b_q40.bin --tokenizer ../tokenizer.bin --weights-float-type q40 --buffer-float-type q80 --prompt "Hello world" --steps 16 --nthreads 64 --workers 10.132.0.3:9998
💡 dim: 4096
💡 hiddenDim: 11008
💡 nLayers: 32
💡 nHeads: 32
💡 nKvHeads: 32
💡 vocabSize: 32000
💡 seqLen: 2048
💡 nSlices: 2
⏩ Loaded 4242882560 bytes
🔶 G 33 ms I 25 ms T 7 ms S 1779278 kB R 522 kB Hello
🔶 G 32 ms I 21 ms T 10 ms S 590 kB R 522 kB world
🔶 G 32 ms I 23 ms T 8 ms S 590 kB R 522 kB ,
🔶 G 33 ms I 24 ms T 8 ms S 590 kB R 522 kB I
🔶 G 36 ms I 25 ms T 10 ms S 590 kB R 522 kB '
🔶 G 33 ms I 20 ms T 12 ms S 590 kB R 522 kB m
🔶 G 34 ms I 23 ms T 9 ms S 590 kB R 522 kB Ben
🔶 G 33 ms I 26 ms T 6 ms S 590 kB R 522 kB ,
🔶 G 33 ms I 26 ms T 6 ms S 590 kB R 522 kB a
🔶 G 34 ms I 27 ms T 6 ms S 590 kB R 522 kB
🔶 G 34 ms I 19 ms T 14 ms S 590 kB R 522 kB 1
🔶 G 34 ms I 23 ms T 10 ms S 590 kB R 522 kB 9
🔶 G 34 ms I 24 ms T 9 ms S 590 kB R 522 kB year
🔶 G 35 ms I 26 ms T 8 ms S 590 kB R 522 kB old
🔶 G 34 ms I 26 ms T 7 ms S 590 kB R 522 kB male
🔶 G 34 ms I 26 ms T 7 ms S 590 kB R 522 kB living
Generated tokens: 16
Avg generation time: 33.62 ms
Avg inference time: 24.00 ms
Avg transfer time: 8.56 ms
n4no_com@instance-3:~/distributed-llama$ sudo nice -n -20 ./main worker --port 9998 --nthreads 64
Listening on 0.0.0.0:9998...
Client connected
💡 sliceIndex: 1
💡 nSlices: 2
⏩ Received 56918016 bytes for block 0 (1237348 kB/s)
⏩ Received 56918016 bytes for block 1 (1264845 kB/s)
⏩ Received 56918016 bytes for block 2 (1293591 kB/s)
⏩ Received 56918016 bytes for block 3 (1211022 kB/s)
⏩ Received 56918016 bytes for block 4 (1422950 kB/s)
⏩ Received 56918016 bytes for block 5 (1422950 kB/s)
⏩ Received 56918016 bytes for block 6 (1459436 kB/s)
⏩ Received 56918016 bytes for block 7 (1422950 kB/s)
⏩ Received 56918016 bytes for block 8 (1459436 kB/s)
⏩ Received 56918016 bytes for block 9 (1459436 kB/s)
⏩ Received 56918016 bytes for block 10 (1422950 kB/s)
⏩ Received 56918016 bytes for block 11 (1459436 kB/s)
⏩ Received 56918016 bytes for block 12 (1459436 kB/s)
⏩ Received 56918016 bytes for block 13 (1422950 kB/s)
⏩ Received 56918016 bytes for block 14 (1459436 kB/s)
⏩ Received 56918016 bytes for block 15 (1459436 kB/s)
⏩ Received 56918016 bytes for block 16 (1422950 kB/s)
⏩ Received 56918016 bytes for block 17 (1459436 kB/s)
⏩ Received 56918016 bytes for block 18 (1459436 kB/s)
⏩ Received 56918016 bytes for block 19 (1422950 kB/s)
⏩ Received 56918016 bytes for block 20 (1459436 kB/s)
⏩ Received 56918016 bytes for block 21 (1459436 kB/s)
⏩ Received 56918016 bytes for block 22 (1422950 kB/s)
⏩ Received 56918016 bytes for block 23 (1459436 kB/s)
⏩ Received 56918016 bytes for block 24 (1422950 kB/s)
⏩ Received 56918016 bytes for block 25 (1422950 kB/s)
⏩ Received 56918016 bytes for block 26 (1459436 kB/s)
⏩ Received 56918016 bytes for block 27 (1422950 kB/s)
⏩ Received 56918016 bytes for block 28 (1459436 kB/s)
⏩ Received 56918016 bytes for block 29 (1422950 kB/s)
⏩ Received 56918016 bytes for block 30 (1459436 kB/s)
⏩ Received 56918016 bytes for block 31 (1422950 kB/s)
Error receiving data: socket closed
32 cpus
n4no_com@instance-1:~/distributed-llama$ sudo nice -n -20 ./main inference --model ../dllama_llama-2-7b_q40.bin --tokenizer ../tokenizer.bin --weights-float-type q40 --buffer-float-type q80 --prompt "Hello world" --steps 16 --nthreads 32 --workers 10.132.0.3:9998
💡 dim: 4096
💡 hiddenDim: 11008
💡 nLayers: 32
💡 nHeads: 32
💡 nKvHeads: 32
💡 vocabSize: 32000
💡 seqLen: 2048
💡 nSlices: 2
⏩ Loaded 4242882560 bytes
🔶 G 34 ms I 26 ms T 8 ms S 1779278 kB R 522 kB Hello
🔶 G 34 ms I 24 ms T 8 ms S 590 kB R 522 kB world
🔶 G 33 ms I 26 ms T 6 ms S 590 kB R 522 kB ,
🔶 G 34 ms I 28 ms T 6 ms S 590 kB R 522 kB I
🔶 G 35 ms I 29 ms T 5 ms S 590 kB R 522 kB ’
🔶 G 34 ms I 26 ms T 7 ms S 590 kB R 522 kB m
🔶 G 34 ms I 26 ms T 7 ms S 590 kB R 522 kB V
🔶 G 34 ms I 23 ms T 11 ms S 590 kB R 522 kB icky
🔶 G 39 ms I 30 ms T 8 ms S 590 kB R 522 kB !
🔶 G 35 ms I 28 ms T 6 ms S 590 kB R 522 kB
🔶 G 37 ms I 27 ms T 7 ms S 590 kB R 522 kB anstalt
🔶 G 35 ms I 29 ms T 5 ms S 590 kB R 522 kB ungs
🔶 G 35 ms I 28 ms T 6 ms S 590 kB R 522 kB management
🔶 G 35 ms I 26 ms T 8 ms S 590 kB R 522 kB .
🔶 G 35 ms I 30 ms T 5 ms S 590 kB R 522 kB ch
🔶 G 36 ms I 28 ms T 7 ms S 590 kB R 522 kB ,
Generated tokens: 16
Avg generation time: 34.94 ms
Avg inference time: 27.12 ms
Avg transfer time: 6.88 ms
n4no_com@instance-1:~/distributed-llama$
CPU Info
n4no_com@instance-1:~/distributed-llama$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 52 bits physical, 57 bits virtual
CPU(s): 90
On-line CPU(s) list: 0-89
Thread(s) per core: 2
Core(s) per socket: 45
Socket(s): 1
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 25
Model: 17
Model name: AMD EPYC 9B14
Stepping: 1
CPU MHz: 2600.000
BogoMIPS: 5200.00
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 1.4 MiB
L1i cache: 1.4 MiB
L2 cache: 45 MiB
L3 cache: 160 MiB
NUMA node0 CPU(s): 0-89
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET, no microcode
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB fil
ling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clf
lush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm cons
tant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid tsc_known_freq pni pcl
mulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx
f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse
3dnowprefetch osvw topoext invpcid_single ssbd ibrs ibpb stibp vmmcall fsgs
base tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed ad
x smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveop
t xsavec xgetbv1 xsaves avx512_bf16 clzero xsaveerptr wbnoinvd arat avx512v
bmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512
_vpopcntdq rdpid fsrm
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Provider: Google Cloud
VM: c3d-highcpu-90 (90 vCPU, 45 core, 177 GB memory) europe-west1, AMD Genoa
Distributed Llama version: 0.1.1
This test used only 64 vcpus or 32 vcpus per VM due to Distributed Llama limitations.
Average Single Token Generation Time
Llama 7B Q40 Weights Q80 Buffer
1 VM
64 cpus
32 cpus
2 VMs
64 cpus
32 cpus
CPU Info
Beta Was this translation helpful? Give feedback.
All reactions