clarify splitting in documentation

[antonkulaga]

I am using embeddings to embed scientific papers. Usually, I use langchain splitters to split the paper into multiple chunks. However, it is not clear to me if fastembed will do splitting for me or I have to split everything (for which I will have to run embedding tokenizer to evaluate tokens per each paragraph).

[NirantK]

FastEmbed will not do the splitting for you. Our default embedding model expects 512 tokens — these tokens are different from the OpenAI tokens!

[antonkulaga]

these tokens are different from the OpenAI tokens!

Of course, I have a custom splitter (in my case https://github.com/longevity-genie/indexpaper/blob/main/indexpaper/splitting.py#L119 ) that computes the tokens with the selected HuggingFace model and splits accordingly. The problem is that with this approach I have to run the embedding model once again - for splitting and I do not win much time. If fastembed would have a tokenaware splitter built-in it will save a lot of computation.

[x4080]

@antonkulaga Yes I thought the same too, is it possible using fastembed to split the texts ?

[antonkulaga]

I think the close is premature. You have to measure number of tokens to split the text and for this you need to call embedding one more. As fastembed does not have proper splitting I will have to use way slower langchain implementation that decreases the benefits from fastembed

[x4080]

@antonkulaga ok, thanks for the answer

[NirantK]

Work in Progress here, we're adding a recursive splitter (based on Langchain, but no dependency) based on tokens: #80

Would appreciate folks sharing any feedback!

[x4080]

@NirantK cool