You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The mp.Queue that we're using to pass the data between the steps is very slow when it's used to send a lot of data (for example when accumulating embeddings from GenerateEmbeddings step for the DeitaFiltering step).
We need to develop a better way (probably using an storage and dumping the content to a file so it can be read back by the other step) to pass large batches, or using a different Queue implementation like this one: https://github.com/alex-petrenko/faster-fifo
The text was updated successfully, but these errors were encountered:
The
mp.Queue
that we're using to pass the data between the steps is very slow when it's used to send a lot of data (for example when accumulatingembeddings
fromGenerateEmbeddings
step for theDeitaFiltering
step).We need to develop a better way (probably using an storage and dumping the content to a file so it can be read back by the other step) to pass large batches, or using a different
Queue
implementation like this one: https://github.com/alex-petrenko/faster-fifoThe text was updated successfully, but these errors were encountered: