Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot change JSONDisk compression level #309

Closed
schance995 opened this issue Mar 25, 2024 · 2 comments
Closed

Cannot change JSONDisk compression level #309

schance995 opened this issue Mar 25, 2024 · 2 comments

Comments

@schance995
Copy link

I tried to change JSONDisk's compression level, but it seems that setting disk parameters at cache initialization doesn't work. To test this, I wrote a script that reuses the JSONDisk example and prints the compression level:

from diskcache import Cache, Disk, UNKNOWN
import json
import zlib

class JSONDisk(Disk):
    """Cache key and value using JSON serialization with zlib compression."""

    def __init__(self, directory, compress_level=1, **kwargs):
        """Initialize JSON disk instance.

        Keys and values are compressed using the zlib library. The
        `compress_level` is an integer from 0 to 9 controlling the level of
        compression; 1 is fastest and produces the least compression, 9 is
        slowest and produces the most compression, and 0 is no compression.

        :param str directory: directory path
        :param int compress_level: zlib compression level (default 1)
        :param kwargs: super class arguments

        """
        self.compress_level = compress_level
        print(f'Init: {self.compress_level}')
        super().__init__(directory, **kwargs)

    def put(self, key):
        print(f'Put: {self.compress_level}')
        json_bytes = json.dumps(key).encode('utf-8')
        data = zlib.compress(json_bytes, self.compress_level)
        return super().put(data)

    def get(self, key, raw):
        print(f'Get: {self.compress_level}')
        data = super().get(key, raw)
        return json.loads(zlib.decompress(data).decode('utf-8'))

    def store(self, value, read, key=UNKNOWN):
        print(f'Store: {self.compress_level}')
        if not read:
            json_bytes = json.dumps(value).encode('utf-8')
            value = zlib.compress(json_bytes, self.compress_level)
        return super().store(value, read, key=key)

    def fetch(self, mode, filename, value, read):
        print(f'fetch: {self.compress_level}')
        data = super().fetch(mode, filename, value, read)
        if not read:
            data = json.loads(zlib.decompress(data).decode('utf-8'))
        return data


c1 = Cache(disk=JSONDisk)
c9 = Cache(disk=JSONDisk, disk_compress_level=9)

print('Storing in c1')
c1['1'] = 'one'
print('Storing in c9')
c1['9'] = 'nine'

print('Retrieving from c1')
c1['1']
print('Retrieving from c9')
c1['9']

The expected output is for c1 to print 1 and c9 to print 9. But instead c9 only prints 9 at initialization and prints 1 otherwise:

Init: 1
Init: 9
Storing in c1
Put: 1
Store: 1
Storing in c9
Put: 1
Store: 1
Retrieving from c1
Put: 1
fetch: 1
Retrieving from c9
Put: 1
fetch: 1

I tried this across python 3.11 and 3.12, and diskcache 5.6 all the way back to 4.0, and all pairs of python/diskcache versions failed to produce the expected result.

This would also affect all subclasses of Disk, since any disk_ arguments will be ignored.

@jakemanning
Copy link

You are using c1 everywhere. If I instead change the access to c9['9'] then I see the expected results

@schance995
Copy link
Author

Thanks for the typo notice. I have reread the docs and found that disk_* should be used to set disk parameters. Initializing c9 as c9 = Cache(disk=JSONDisk, disk_compress_level=9) gives the correct result.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants