Roadmap (2024)

[westonpace]

SDK

Rust

Description

The following is a rough roadmap of things we would like to tackle this year. There are no fixed dates and the roadmap is flexible, but it is a general indication of what people are working on. The roadmap is in rough priority order (higher priority items at the top) but outside the top items (which are actively being worked on) this is pretty fluid.

Unified API

Currently the rust, python, and node APIs are all completely independent. As a result, there is considerable divergence in available features. Python is the most complete, with support for things like hybrid search, and rust is the least complete, as it doesn't even have embedding support. We would like to migrate all SDKs onto one common (rust) base.

• LanceDB Async API epic #926
• Feature: complete rust API #996

New Functionality

ML workloads need unique operations that aren't part of traditional database workflows. We have added support for those in the past (embeddings, hybrid search) and want to continue exploring what capabilities we can offer.

• Support for fine-tuning
• FTS that works on cloud storage, supports incremental indexing

Client / server model

LanceDb currently runs embedded in the user's application. This lends itself to simpler architectures and solutions. However, there are some cases where a client / server model can be useful. For example, if the client is outside the datacenter or to develop ad-hoc clients in new languages that don't have rust support.

• Epic: Simple remote server implementation #1118

Easier Ingestion

Adding data into LanceDb properly can be complicated. Indices need to be updated, compaction needs to be run, inputs need to be batched. The default settings for LanceDb should make this easier.

• Epic: Automatic Maintenance #1115
• Epic: Simple add batching #1116
• Epic: clustered indices #1119
• Feature: Row IDs #1120
• feat: enable dynamodb commit in python sdk #534

Query API

Currently it is possible to plug LanceDb into an existing query engine (duckdb / datafusion / etc.) However, we can also pretty easily expose the underlying datafusion query engine as an SQL frontend to give users the ability to run SQL statements without needing another dependency.

• SQL frontend
• Table views