Does the SPU means a node in cluster?

[whoconli]

1. What is the relationship between SPU and node(or a device)? If I have 3 computers in my cluster, how many spus may exist? 3 or more?
2. If I have a topic with 3 SPU, one is leader, others are follower, do they process the data stream in parallel? Or only leader process, others are only for fault tolerance？ Because I don't see the data synchronization in the picture

[sehz]

Hi, Great questions.

1. Usually, you should deploy 1 SPU per compute node (VM). You need at least 2 SPUs to guarantee some reliability, with three as a good min number of nodes.
2. Yes, follower exists mainly for fault tolerance. All read/writes go thru the leader. If you want to process perform concurrent stream processing, partitioning is the way to do that.

[whoconli]

Thanks! And I have another question. Does fluvio treat the SPUs as peers (followers may be selected as leader)? Does fluvio consider the heterogeneous node? Actually, different nodes may have different resources with different performance. For example, if we have 2 SPU, one is 1C2GB, and another is 16C32GB. With the assignment algorithm, the data will be processed by these SPUs in an equal way. This may cause resource overload for 1C2GB and low utilization for 16C32GB.

[sehz]

Using custom SPU, you can assign different node types since you will be responsible for provisioning. For SPU Group, since it is provisioned in Kubernetes, they will be assigned the same nodes.

However, our current topic resource algorithm assumes homogenous node types. In the future, we could support more custom resource allocation using SmartModule.

[whoconli]

That is to say, in a topic, all the SPUs are the same, cause they are managed by k8s (in the form of deployment)? And SPU actually is a VM (or container?), providing wasmtime runtime to run wasm and process the data?

[sehz]

SPU is just a process. It can be deployed in a container, pod, or VM.

[whoconli]

Thanks!

But it confuses me that SPUs run in pods (container or VM), do the relative smartmodules also run in the pods?

And when I create a new topic with 6 partitions, fluvio will create 6 pods to deploy SPU in the cluster?