Tiered Storage

[rupurt]

Hi,

Do you have any plans to implement tiered storage out to the likes of S3/R2/MinIO?

[sehz]

Hi Alex,

Good question. This is one of the potential items in the roadmap. What would be your preferred storage for the first implementation?

[rupurt]

S3 makes sense to me because that's what most people use. However if it's S3 compatible it theoretically "should just work" 😄 with R2 & MinIO.

[ajhunyady]

@rupurt would a connector for In/out of S3 be a good starting point? I seems like something a bit different from what you are asking, but it may solve your use case while we are implementing S3 in the engine.

[rupurt]

@ajhunyady it would be handy but it's not what I'm looking for.

To me tiered storage is about fully realizing the Kappa architecture and using the real time processing system as the system of record. For that to work on a large team with different philosophies you really need a simple and consistent programming model. Having the real time processing system abstract away the complexities of long term storage and reprocessing behind topics is a huge win.

[ajhunyady]

Thank you for the additional clarification. It makes perfect sense.