Replies: 3 comments
-
the logic is in https://github.com/seaweedfs/seaweedfs/blob/master/weed/shell/command_ec_encode.go the constraint is that when paralleling the encoding, the encoding server should not try to encode multiple volumes at the same time. need some careful coding on this. |
Beta Was this translation helpful? Give feedback.
-
Understandable, because you definitely don't want to lose data. I did some tests on our 8 server cluster with RAID0 raid sets. When I create a RAID0 with 6 HDD drives on a volume server with 90 drives, I get 15 RAID0 drives. This brought the encoding time back to 8 minutes instead of the 25 minutes. I use the following configuration;
The maintenance script that runs every 17 minutes:
It would be very help full if the encoding goes faster, do you have any tips what I can do so speed it up more? |
Beta Was this translation helpful? Give feedback.
-
The Reed Solomon library used mentions to build it with |
Beta Was this translation helpful? Give feedback.
-
On our current cluster we store around the 250TB per day. We do this with replication of 010 and a copy_2 = 720, this combination works very well.
But we are seeing issues when the erasure code starts (ec.encode fullPercent=95 -quietFor=1h). It can't keep up with the data that is written to the cluster. It takes around 25 minutes per volume to encode and distribute the data.
Is it possible to make this faster? It would be nice to have an option to run it multi threaded with say x threads per server.
To minimise the performance impact, it would be great if compactionMBps is also working for the ec.ecode.
Best regards,
bvanelst
Beta Was this translation helpful? Give feedback.
All reactions