-
Notifications
You must be signed in to change notification settings - Fork 175
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: replicate boundary vectors to multiple partitions #2258
base: main
Are you sure you want to change the base?
Conversation
Signed-off-by: BubbleCal <bubble-cal@outlook.com>
…e-boundary-vectors
@@ -106,6 +113,80 @@ impl<T: ArrowFloatType + L2 + Dot> IvfTransformer<T> { | |||
|
|||
UInt32Array::from_iter(result.iter().flatten().copied()) | |||
} | |||
|
|||
/// Compute the partition for each row in the input Matrix. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we use this in search? or just in build time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
only in build time
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just one concern: can we define compute_partitions
in terms of compute_multiple_partitions
instead of duplicating the code?
same for compute_multiple_memberships
let part_ids = self.compute_partitions(&mat).await; | ||
let field = Field::new(PART_ID_COLUMN, part_ids.data_type().clone(), true); | ||
Ok(batch.try_with_column(field, Arc::new(part_ids))?) | ||
let part_ids = self.compute_multiple_partitions(&mat).await; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmm, maybe we should use a different transformer for search?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we use transformer for search?
the code path should be IVFIndex::find_partitions -> IvfImpl::find_partitions -> kmeans::find_partitions
right?
sure we can, just duplicated the code for now because |
No description provided.