Embeddings management

[andered]

My apologies in advance if this is a silly question - I'm new to Python and still learning some of the basics. My question: Is there a simple way to save the embeddings aside from using Pickle files? I'd like to pre-process a very large library (hundreds or perhaps 1000 scientific papers), store the embeddings somewhere, and then select small groups for targeted Q&A. Re-processing each time is both expensive and slow. (I see that there's supposed to be some sort of built-in caching, but this doesn't seem to work for me or I'm not using it properly.) Thanks in advance for any guidance you may be able to provide.

[whitead]

The most direct way is pickling. You can also store them externally in a database and rebuild using the add_texts function directly, which takes in the text and embeddings (which you can store in a CSV file or database).

[andered]

Great - I appreciate the guidance. I'll go through your code and see how I might be able to do this. Thanks!