Welcome to FASTER Discussions!

[badrishc]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[AshK14]

Hi,

Thanks for creating this library! I am new to using it so would like some recommendations:
We are using FasterKV and taking HybridLogCheckpoints at an interval of 10 seconds. and Index Checkpoints at a frequency of 2 minutes.
The problem I see is that the total memory footprint (calculated by: store.IndexSize * 64 + store.OverflowBucketCount * 64 + store.Log.MemorySizeBytes) remains constant at about 135 MB but the application that uses FasterKV is fluctuating between 400MB - 1GB of RAM (no other heavy operations within the application itself).

Could it be because of the frequent checkpoints that we are taking?
If so, what could be the recommendation on checkpointing frequency.
Also, would be great if you could suggest any other areas to look at for this fluctuating memory issue.

var logSize = 1L << 20;
this.log = Devices.CreateLogDevice($"{this.dataFolder}\\data\\Store-hlog.log", preallocateFile: false);
objLog = Devices.CreateLogDevice($"{this.dataFolder}\\data\\Store-hlog-obj.log", preallocateFile: false);

this.db = new FasterKV
    <string, DeviceValue>(
        logSize,
        new LogSettings
        {
            LogDevice = this.log,
            ObjectLogDevice = this.objLog
        },
        new CheckpointSettings
        {
            CheckpointDir = $"{this.dataFolder}/data/checkpoints",
            RemoveOutdated = true
        },
        new SerializerSettings<string, DeviceValue>
        {
            valueSerializer = () => new CustomTypes.MyValueSerializer()
        }
    );

Above is the store that we are using.

We also use
.Iterate() to iterate over the keys and after consuming the value in our app, use .Delete() within the same session.
Upsert is happening in a separate session.
These 2 sessions are not disposed until the windows service is stopped.

[badrishc]

Since you have C# class objects (not just blittable structs) as keys and values, they are stored on the heap and not included in the size computation above. Are you accounting for the space occupied by the key and value itself (string and DeviceValue)? To see how to control the total footprint including heap objects, see CacheSizeTracker in this sample: MemOnlyCache. I don't see offhand why checkpointing should cause memory increase. Read more docs on memory utilization here as well.

[AshK14]

Hi,

Thanks for your reply. The CacheSizeTracker seems to be a good starting point to manage objects on the heap.
I had some further doubts. I read from the documentation that PageSizeBits has a default of 25 which is 2^25.
So what would be the default value for MemorySizeBits if we don't explicitly specify it?

Also, what would be the typical CPU usage when using FasterKV with frequent HybridLogCheckPoints (at a frequency of 10 secs) and IndexCheckpoint (at a frequency of 2 minutes).
I am noticing that it takes up around 3% to 4% when used on a VM with the below configuration:

Also this amount of usage is happening when there are no Upsert() and also there does not seem to be any data in the log since I've ensured that the objects are deleted in FasterKV log after they are consumed by the application. There is however .Iterate() going on every 0.5 seconds.

Just wanted to understand if this kind of CPU usage is expected.

Again, thanks for your time.

[AshK14]

I've boiled it down to this piece of code which is causing both spike in memory and CPU

            using (var prc = fasterDbStore.db.For(new Functions()).NewSession<Functions>())
            {
                while (!threadAborted)
                {
                    try
                    {
                        using (var iter = prc.Iterate())
                        {
                            while (iter.GetNext(out var recordInfo) && !recordInfo.Tombstone)
                            {
                                .....
                            }
                         }
                       }
                    }
                 }

The above piece of code is on a background thread and runs every 0.5 secs.
Is there any way I can find out if there are any records in FasterKV log and only if so do the .Iterate() ?
I would not want to iterate the FasterKV log when there are no records in it.

Thanks

[badrishc]

This code seems to continuously iterate through the log over and over, which will be expensive. To catch the case where the store is empty, just look for store.EntryCount. Also, if no new items were added to the DB, you might not want to perform the iteration again. You can do that in your code outside FASTER: whenever you upsert, set a dirty bit. And perform another iteration only if the dirty bit is set (reset it before starting the iteration).

[roozbehid]

I am a bit fuzzy about use cases of this project.
Should it be looked like as potential rival of Redis? Should it be used for like logging in your application?
I think it helps to better describe project with potential use cases and also how it compares to some of similar projects.

[badrishc]

FasterKV

• Embedded dictionary (key-value store) that is also persistent and supports data larger than memory
• Supports remote access as well, via FASTER.remote
• Useful as a database, cache, dictionary, etc.

FasterLog

• Persistent logging library
• Similar functionality as Kafka, provided via C# at high performance saturating modern drives.

[AshK14]

Hi Badrish,

In continuation to our discussion above, I had a doubt with object log file that spans across memory and disk.
Everything works great for me (Upsert, Iterator and Delete).
After I do the session.Delete() operation, I don't see the data coming back through the session iterator which is what I expect.
I was under the impression that session.Delete() would also remove the record from memory and disk.
But it looks like my log files sizes have increased a great deal,

So my question would be, why does the session.Delete() not remove these records since I no longer need them. I was also reading about Log Compaction. Is that the way forward for me to remove these records? If so, please advice if something like below will make sure that I remove only the records that I have deleted through session.Delete()

long compactUntil = store.Log.BeginAddress + 0.2 * (store.Log.SafeReadOnlyAddress - store.Log.BeginAddress);
compactUntil = session.Compact(compactUntil, shiftBeginAddress: false);

Thanks again for your help.

[AshK14]

@badrishc Can you please have a look at my question above?
Also, how can I tell log compaction to make sure that the records that have been deleted through a session.Delete() alone have to be removed from the object log file?

Thanks

[badrishc]

session.Delete only marks the record as deleted, or puts in a tombstone marker on the log. The record itself is not removed from disk. This is because the log is log-structured. The way to get rid completely of tombstoned records is to perform log compaction, as you rightly figured out from the documentation.

Log compaction will not delete normal undeleted records. It will move LIVE records from the head of the log to the tail. After it is done, you can call ShiftBeginAddress to truncate the dead region of the log (from old begin address until compactUntil address). See our sample at https://github.com/microsoft/FASTER/tree/master/cs/samples/StoreLogCompaction for details.

[AshK14]

@badrishc I have now implemented Log Compaction and have recorded the below observations.
The object log and memory still seem to be increasing but at a slower rate than before compaction. I believe that the compaction is working since I am logging the results like below:

2021-11-15 20:18:26,821 UTC - FasterDb - Compacting log since BeginAddress is : 271099000 and SafeReadOnlyAddress is: 301510000, will be compacted until address: 298468900
2021-11-15 20:18:27,600 UTC - FasterDb - Compacted log until address: 298468912
.
.
.
2021-11-15 20:21:27,978 UTC - FasterDb - Compacting log since BeginAddress is : 301509976 and SafeReadOnlyAddress is: 301510000, will be compacted until address: 301509997
2021-11-15 20:21:28,019 UTC - FasterDb - Compacted log until address: 301510000

Above I am logging store.Log.BeginAddress, store.Log.SafeReadOnlyAddress and compactUntil and the compaction is done using the code provided in the documentation

  long compactUntil = store.Log.BeginAddress + 0.9 * (store.Log.SafeReadOnlyAddress - store.Log.BeginAddress);
  compactUntil = session.Compact(compactUntil, shiftBeginAddress: false);
  await store.TakeHybridLogCheckpointAsync(CheckpointType.FoldOver);
  store.Log.ShiftBeginAddress(compactUntil);
  await store.TakeHybridLogCheckpointAsync(CheckpointType.FoldOver);

Only difference being, I am compacting 90% of the logs and this runs on a separate thread every 30 seconds when there is no activity (Upsert/Delete) happening in Faster which is decided based on a threshold (have handled this by measuring the time between now and when the last upsert/delete in FasterKV took place).

Every time, I do the following sequence of steps in the code:

• Upsert-Read-Delete for 4 hours (Upserting 100 records per second into FasterKV)
• After 4 hours, wait for a threshold to make sure that there is no activity on Faster (say 3 mins),
• Compact the log

I have repeated this for 12 times now.
Initially after the first run, the Object Log size was 167MB and Memory was around 1063MB.

Finally now, I have Upserted/Deleted close to 5M records overall, and my Object Log size is 1.19GB and Memory is around 3GB.

Other note is that I am not Disposing the Faster objects unless the service is stopped, since my application is supposed to be running continuously. The service was not stopped for the entire duration of this test.
Also, I am using the default settings when creating the FasterKV object.

Can you please advice me on why the object log and memory sizes increase linearly even though I compact the Faster log every time before the next load of records (for 4 hours) come in.

Thanks for your help as always!

[badrishc]

What you are doing seems fine. Note that when using objects, the default page size might be way too large, as it can hold a humongous number of objects (since only 8 byte refs are stored on the main log). Please reduce page size (via LogSettings) to say 4KB or 16KB (bits = 12 or 14 respectively) and see how things work. Memory size also should be designed accordingly (see tuning docs).

[badrishc]

Try running and modifying our sample here: https://github.com/microsoft/FASTER/tree/master/cs/samples/StoreLogCompaction -- if you see that is not exhibiting the same memory behavior, then it must be something about the way you are doing things differently from the same. You can pay attention to the log constants in the sample as well.