Protect Against Repeated KeyboardInterrupt Signals

[rmorshea]

Summary

Right now, if a user sends repeated KeyboardInterrupt signals it's possible for diskcache to get stuck in a locked state because the cleanup logic that's meant to unlock the cache never gets executed.

Use Case

In my case I'm writing software used by scientists who have a habit of holding down Ctrl+C or press it when they want their program to exit. Sometimes this causes diskcache to hang the next time they run their program since the cache is stuck in a locked state.

Workaround

My current workaround is to expire everything in the cache when the program starts:

if mp.parent_process() is None:
    # Forcefully expire all cache entries to ensure we're starting fresh.
    # This only deletes 100 records at a time so we need to loop until the
    # deleted count returned by expire() is 0. This also will not evict
    # anything that does not have an expire time set.
    while DISK_LRU_CACHE.expire(now=sys.float_info.max):
        pass

This has several downsides though:

• expire() is only available on Cache and not FanoutCache
• This prevents values from being cached across runs of the program
• If one instance of the program is currently running, kicking off another will expire values currently in use by the other.
• Adds some (though minimal) overhead at the start of the program

Solutions

The only solution I can think of at the time of writing is to defer interrupt signals until after diskcache has relinquished control of any resources it acquired. This thread suggests a way to do this. Currently there's no way to do this myself without calling lower level diskcache APIs and give up on using any diskcache.recipes.