Randomness of MIlvus Collection

[Chahnwoo]

I am working with the LangChain integration of Milvus and wanted to ask about how the Milvus HNSW algorithm works. Is there any degree of randomness introduced at any point of graph construction or search?

Here is a general overview of the collection generation process that I am using:

1. Retrieve a list of texts
2. Iteratively upload those texts into the MILVUS collection with a pre-defined embedding model

I am not using any randomization in the insert process and have been using the same embedding model and question set for all tests, yet seem to be getting different search results each time I create a new collection. Does anyone know what the problem might be?

[yhmo]

Some questions:

1. How many texts are uploaded to the Milvus collection?
2. Did you specify the "index_params" when you initialize the Milvus VectorStore?

[Chahnwoo]

Thank you for the answer! Can I take that to mean that there is no randomness introduced during the search process, the way it is during index building?

[yhmo]

Milvus splits data into segments, each segment has an independent index. Segment size could range from 100MB+ to 1GB+. Search engine searches topk results from each segment, and merges N topk to be a final topk result. So, different segment sizes slightly affect the search result. Segment size is affected by insert actions(batch by batch or row by row, insert interval, etc).
Index algorithm also affects search results.

[Chahnwoo]

But once the vector database has been fully created (no new inserts or deletions or updates take place), those same indices are used consistently for search, right? Assuming no modifications to a created vector database, there is no randomness introduced to search...?

[yhmo]

I mean two different full test processes might get different segment size.
Let's say we want to test 1M entities with different index types.
Process 1: create collection A, insert data batch by batch, 10000 entities for each batch. create index, test search
Process 2: create collection B, insert data batch by batch, 500 entities for each batch. create index, test search

Assume the total data size is 500MB.
The process 1 might get these segments: 130MB + 130MB + 130MB + 110MB.
The process 2 might get these segments: 120MB + 120MB + 120MB + 140MB

Data distribution is different. Search result might be a bit different.

If you use the same collection to test, no additional data inserted. Yes, segment size is not a problem.

[Chahnwoo]

Thank you so much! Your answers have been so helpful.