Any way to perform faster log truncation based on time?

[vangarp]

Is there a way I can truncate entries from fasterlog based on time when they were added? We can truncate based on size, but any approach we can take to truncate any entries that were added before some time etc? Any interfaces I might be able to implement to get a functionality like this?

[badrishc]

FasterLog places items sequentially on the log, based on ingestion time. If I understand you right, you want to truncate all records ingested say > 60 minutes back. One option to do this is to keep an array of say 12 TailAddress values, one element for every 5 minutes, for a total of 60 minutes. Every 5 minutes, you remove the oldest element in the array (say OldTailAddress), and replace it with a new element with value = current TailAddress. (The array is used like a circular buffer.) You can then truncate FasterLog until OldTailAddress. You can change the constants used here based on your specifics.

[vangarp]

That sounds exactly what I want. Let me try that out

[vangarp]

@badrishc your approach works great when application is running. Anything we can do to handle something similar after process restarts? Like if I have written something at time X and my process restarts, can I know that the record written at time X is now too old and so I can truncate it? I am ok if the answer is that "you will need to do that book-keeping somewhere else and this is beyond the scope of faster" or if there is some other interface I can implement where I can have such logic in. Thanks again!

[badrishc]

FasterLog v2 during commit can take in a cookie that you can access after restore. You could use this to store away such extra information. In v1, it's still doable, but a little more involved. You can extend DeviceLogCommitCheckpointManager to add more metadata to the commit.dat file.

[badrishc]

Here is a bit more detail on this commit manager approach, when using v1:

1. Create your custom commit/checkpoint manager, extending DeviceLogCommitCheckpointManager
2. Override Commit method, to add your custom data in addition to the standard
3. Peel off your extra data during recovery by overriding the GetCommitMetadata method
4. Use this custom commit manager when instantiating FasterLog, via FasterLogSettings.LogCommitManager

[vangarp]

Perfect! That’s the route I was going, but great to get a confirmation from you too