X86 VM + NVIDIA Jetson Nano 4GB + 1Gbps Switch [Llama 2 7B/13B]

[JiangShanCode]

Devices

• VM (X86): 10cores, 24GB Mem
• Jetson Nano (ARM): Cortex-A57(4cores), 4GB Mem
• Network: Ruijie RG-S2910-24GT4XS-E Switch (1Gbps)

Implementation

• Version: 0.1.1
• Root node: VM (Unfortunately, if I set Jetson Nano as root, it crashes due to insufficient memory.)
• Worker node: several Jetson Nano
• Weight: Q40, Buffer: Q80, nthread: 4

Results

"-" denotes out of memory.

llama-2-7b

Metric	1VM	1VM(nthread 8)	1Nano	1VM+1Nano	1VM+3Nano	1VM+7Nano
Avg Generation Time (ms)	200.81	178.14	-	885.11	540.42	372.31
Avg Inference Time (ms)	200.44	177.65	-	135.36	96.49	78.83
Avg Transfer Time (ms)	0.13	0.15	-	749.48	443.60	293.21

llama-2-13b

Metric	1VM	1VM(nthread 8)	1Nano	1VM+1Nano	1VM+3Nano	1VM+7Nano
Avg Generation Time (ms)	370.56	327.65	-	-	961.32	619.30
Avg Inference Time (ms)	370.17	327.20	-	-	153.35	120.72
Avg Transfer Time (ms)	0.15	0.19	-	-	807.82	498.32

The detailed log can be found.

Discussion

Thank you sincerely for your efforts in developing this fascinating project. I have a few inquiries that I would be grateful to discuss:

• I have observed in my experiments that the transmission latency decreases with an increase in the number of nodes, which seems to contradict previous examples (such as the Raspberry Pi 4B). Is this normal?👀
• Additionally, I'm curious if there are any plans underway to develop GPU support :).

[b4rtaz]

Hello, thanks for results. It's awesome to see the performance of Jetson Nano.

I have observed in my experiments that the transmission latency decreases with an increase in the number of nodes, which seems to contradict previous examples

It's weird. I suppose maybe this is some problem with the router. All my observations show the opposite pattern: more devices = slower transfer ([1], [2]).

Additionally, I'm curious if there are any plans underway to develop GPU support :).

Yes, there are. But it's difficult to say something more now.

I'm wondering, have you thought to test only on Jetson Nano? The VM inside your reports introduces a bit of ambiguity. Because the VM may be much much faster than Jetson Nano. Jetson should handle 7B model on 2 devices I think.

[JiangShanCode]

I attempted to use Jetson Nano to handle a 7B model on two devices, but it crashed. The Nano designated as the root node consumed the majority of the memory, leading me to suspect that there was insufficient memory available (only 4GB). Unfortunately, I do not have Jetson Nano with larger memory capacity available. So, I set a VM as root and used Jetson Nano as a worker.😂

Root node:

Work node:

Memory consumption of the root node:

[JiangShanCode]

It works on eight devices. Here are the results and logs.

llama-2-7b

Metric	8 x Jetson Nano 4GB
Avg Generation Time (ms)	449.50
Avg Inference Time (ms)	367.76
Avg Transfer Time (ms)	80.29

I think the inference time should be determined by the weakest node (i.e., Jetson Nano) in the cluster because distributed-llama evenly distributes the computational tasks across all nodes. So I expect the latency of two configurations (8Nano and 1VM+7Nano) to be similar. However, the inference time of the 8 Nanos increases significantly compared to 1VM+7Nano (from 78.83ms to 367.76ms). Besides, according to the logs, the bandwidth per worker in 8Nano is lower than 1VM+7Nano (from 16820 kB/s to 7017 kB/s, and all devices are connected by 1Gbps switch), but the transmission time decreases (from 293.21ms to 80.29ms ). Is there anything I have misunderstood?

[b4rtaz]

I think the inference time should be determined by the weakest node

Not exactly. The root node has a bit more to calculate than workers, because not all layers are distributed. This is a reason why the root node requires a bit more RAM memory than workers.

About the transfer time, the transfer time is calculated after the root node finished processing the assigned fragment of a layer. Basically, it's a time to get all missing outputs from other fragments. If the root node is much faster, it will be waiting longer (so the transfer time will increase). If you have the same devices, then the transfer time is very close to the real transfer time because all devices have the same number of calculations to be performed.

[JiangShanCode]

OK, I get it. Thank you!😀

[JiangShanCode]

By the way, does the inference time also represent the latency from the start to the completion of the specified layer segment by the root node, rather than the average computation time across all nodes?

[b4rtaz]

Yes, this is the inference time of the root node.