FasterKv logs and NTFS compression

[avonwyss]

We use FasterKv successfully for a LRU cache and we have significant amounts of data stored on disk. The data stored is mostly very small chunks, so that compressing these individually does not yield good results. However, many of the chunks are very similar, thus compression of our Faster logs would be quite efficient in terms of space: On a small test instance we have compressed 16.5 GB to 2.5GB just using NTFS compression.

We wonder, however, if this is something which would be OK to be used in production or not? Are there issues to be expected?

[badrishc]

It should be fine to use any compression. We have users in production that add CRC etc. at the IDevice level, so I don't see why compression would be any different. You just need to adhere to the API precisely.

[avonwyss]

Thank you @badrishc for the reply. Note that the question is about using an external compression on the file system level, therefore this is independent of the API. See for instance SQL Server where this is not allowed, so is there a similar reason not to use NTFS compression.

[badrishc]

I see, thanks for the clarification. On Windows, FASTER uses LocalStorageDevice by default, which uses sector aligned writes (via windows file APIs, overlapped asynchronous IO) which may not work with NTFS compression. However, you can instead use ManagedLocalStorageDevice as your device, that should be fine as it simply uses standard .NET file APIs. This is the device used by default on Linux as well.

[avonwyss]

Thank you @badrishc. I looked at the code for LocalStorageDevice and I did not see anything which would prevent NTFS compression from working; I could only find normal async (overlapped) file IO calls which are fully supported in NTFS compression AFAIK. So if that is the only concern then I assume that compression is fine then.