[s3] recursive list of keys on filer store with SQL

[kmlebedev]

What problem are we solving?

The registers contain a large number of nested keys, which leads to exponential growth of the query list for each subdirectory.

How are we solving the problem?

Iterating over keys by prefix on the SQL side of the database

How is the PR tested?

S3TEST_CONF=s3tests.local.conf tox -- s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_basic

all list v2 tests

S3TEST_CONF=s3tests.local.conf tox -- 
s3tests_boto3/functional/test_s3.py::test_bucket_listv2_many
s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_basic s3tests_boto3/functional/test_s3.py::test_bucket_listv2_encoding_basic s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_prefix s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_prefix_ends_with_delimiter s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_alt s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_prefix_underscore s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_percentage s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_whitespace s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_dot s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_unreadable s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_empty s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_none s3tests_boto3/functional/test_s3.py::test_bucket_listv2_fetchowner_notempty s3tests_boto3/functional/test_s3.py::test_bucket_listv2_fetchowner_defaultempty s3tests_boto3/functional/test_s3.py::test_bucket_listv2_fetchowner_empty s3tests_boto3/functional/test_s3.py::test_bucket_listv2_delimiter_not_exist s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_basic s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_alt s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_empty s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_none s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_not_exist s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_unreadable s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_delimiter_basic s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_delimiter_alt s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_delimiter_prefix_not_exist s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_delimiter_delimiter_not_exist s3tests_boto3/functional/test_s3.py::test_bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist s3tests_boto3/functional/test_s3.py::test_bucket_listv2_maxkeys_one s3tests_boto3/functional/test_s3.py::test_bucket_listv2_maxkeys_zero s3tests_boto3/functional/test_s3.py::test_bucket_listv2_maxkeys_none s3tests_boto3/functional/test_s3.py::test_bucket_listv2_unordered s3tests_boto3/functional/test_s3.py::test_bucket_listv2_continuationtoken_empty s3tests_boto3/functional/test_s3.py::test_bucket_listv2_continuationtoken s3tests_boto3/functional/test_s3.py::test_bucket_listv2_both_continuationtoken_startafter s3tests_boto3/functional/test_s3.py::test_bucket_listv2_startafter_unreadable s3tests_boto3/functional/test_s3.py::test_bucket_listv2_startafter_not_in_list s3tests_boto3/functional/test_s3.py::test_bucket_listv2_startafter_after_list ```
# Checks
- [ ] I have added unit tests if possible.
- [ ] I will add related wiki document changes and link to this PR after merging.

[chrislusf]

• The change is internal to the filer store backends. The allowListRecursive option can be avoided. With the data layout change, it is basically a different filer store.
• The PR is too big to review.

[kmlebedev]

• The change is internal to the filer store backends. The allowListRecursive option can be avoided.

Hi, I wanted to leave the allowListRecursive option for some time, as an experimental one, since the tests do not cover all possible cases and might break something for someone.

• With the data layout change, it is basically a different filer store.

I don’t understand what changes we are talking about here, but the data structure in the stores does not change, perhaps indexes will need to be added for SQL.

• The PR is too big to review.

It is large mainly because of the plug functions for each filer. But I will try to reduce it as much as I can by moving the new Ceph S3 tests fixes into a separate PR and adding new github actions tests.

[chrislusf]

This would need a secondary index on directory+'/'+name, which can be costly for large directories.