Roadmap 2024

[wjones127]

Vector search

We continue to work at making vector search performant at very high scales (10 billion vectors).

• Full support for fp16 and bf16 vector indices #1832
• IVF HNSW Indices #2078
• Implement vector clustering based on BFS HNSW order #2064

V2 File Format

We are working on the next version of our Lance file format. The key improvements are null support in all types, columnar compression, and better performance with large blobs.

• Lance File Format Version 2 (technically v0.3) #1929
• feat: null support for plain encoding #1643

Table format

We want to make table maintenance automatic, if opted into. This would make it much easier to manage tables, since you would not need to manually call operations like compact, re-index, and cleanup.

• Auto-{compact, cleanup, re-index} #2061

Full text search

We will work on native full text search indices for Lance. LanceDB's current implementation of FTS doesn't support incremental updates or working in Node or Rust. Nor does it work on object storage. Implementing FTS at the Lance level will remove these limitations.

• Full text search (FTS) indices #1195

Read / write integration

To make Lance a great choice for AI data lakes, we will be adding connectors to two of the most popular distributed frameworks, Spark and Ray.

• Spark
  • Add Spark data source for lance #541
  • Create Spark writer #2062
• Ray connector for Lance #2063

Query

Finally, there are smaller features we'd like to implement to improve queries. Providing a general SQL front-end for SELECT queries will provide native support for aggregations as well as provide an easy way to implement table views. Bitmap scalar indices will provide much better filter performance on columns with few distinct values.

• Add a sql method to execute arbitrary SELECT statements
• Register user-defined functions for use in projections and filters
• Table views (dynamic columns)
• Bitmap scalar index Store a compressed bitmap alongside the index pages #1552
• More flexible query plans Expose DataFusion extensions (plan nodes, plan optimizer rules) #1782
  • Filter by _distance column _distance filter pushdown #2096