Support pandas for users without a database set up

[thejumpman2323]

Checkout for more info
#636

df = pandas.read_csv('data.csv')

df = superduper(df)
df.add(model)

df.execute(...)

[blythed]

For this we would need to implement the "filesystem" metadata and artifact stores (easy).
This could potentially serve as a simple test of how easily extensible our patterns our.

[thejumpman2323]

so you are suggesting filesystem will be used as our database to store models, outputs etc

[thejumpman2323]

@blythed What if we utilise a lightweight duckDB or sqlite for this purpose

can we have a pro and con of both?

@blythed

cc
@rec @nenb @kislerdm

[kislerdm]

As per our discussion with @thejumpman2323 today, Kartik and I feel that there is value worth the effort investment:

• pandas.DataFrame object is an "API" which can implement the SDDB "data interface" (e.g. DataSet) used by the Model.
• Metastore is an independent problem which will be addressed in line with the initiative to make SDDB extensible for supporting virtually any database, including file system, or in-memory dbs.

lightweight duckDB or sqlite

I feel that it's definitely worth assessment, but it's beyond the DataFrame support. Technically, our users don't need database at all to benefit from SDDB: they shall be able to use the models and vector search without persisting model and its config on disk.

[nenb]

(I've just read through #636 and will leave comments here.)

I think this is an absolutely essential use-case for superduperdb, well done for starting the discussion on this!

I think the tech-stack that you mentioned (ibis, arrow, sqlite, duckdb) aligns well with the initial users that we would like to support - single devs or small teams who want a Python-first interface. The nice thing about ibis is that it supports all of these packages (and many more) as backends. I think it would be very useful to do an initial spike (ie a time-boxed exploration) of ibis to see if it aligns with how superduperdb is designed. If it works out, it could save us a lot of work, and provide support for pandas users (and many others) quite quickly.

In terms of using sqlite (or alternatives) as a metadata store, and a local filesystem for blob storage, this is a common pattern for a bunch of AI projects already it seems. EvaDB have adopted this pattern for example. I think the only opinion I have here is that we should try and provide a Python-first interface to users if we implement this feature eg I can execute commands like .read_csv() or .filter() rather than relying on SQL knowledge for commands like like .execute("SELECT * FROM ..."). Requiring SQL knowledge is perfectly okay (EvaDB do this for example), but my feeling is that we would like to make our package as approachable as possible to everyone in the Python community, even if they aren't familiar with SQL.

Nice work! 🚀

[blythed]

Agree to attempt to handle pandas with ibis framework, in order to reduce work. See discussion #643.

[blythed]

Solved in #668