How to create a bucket in Minio S3 with a minimal race window?

[dstromberg]

Hi.

We've been using Minio S3 and have been happy with it.

Now that we have some of our code in Kubernetes (we used to just use docker-compose), we're finding that sometimes we get:
botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: The specified bucket does not exist

...and we think that's probably due to a race condition, related to Kubernetes starting a bunch of things at once and wanting each container to sort out its own retries and service dependencies.

My question for this forum is: Is there a way of creating a bucket an instant after Minio S3 comes up, as part of Minio S3's bootstrap process? Our software always uses the same bucket name and creds for a given environment.

Things seems to work in docker-compose (race-condition free), because of the nice dependency management docker-compose provides.

We currently create the bucket with a script I wrote (create-permissive-bucket) that mostly just does the following. This identical script is used for docker-compose and Kubernetes:

aws s3api create-bucket --bucket "$bucket_name" --region us-west-2 --endpoint-url="$endpoint_url"

And:

aws \
        s3api \
        put-bucket-policy \
        --bucket "$bucket_name" \
        --region us-west-2 \
        --endpoint-url="$endpoint_url" \
        --policy file://"$tempfile"

If we run that script at the beginning of another container command that depends on s3minio being healthy (using docker-compose), things appear to work consistently.

Our Kubernetes setup is done without any (service) dependency management, so sometimes the create-permissive-bucket finishes later than we'd prefer.

Does s3minio provide a way to create a bucket right away after s3minio comes up? I'm thinking this would greatly reduce the race window.

I know, it's probably a good idea to make sure our S3 clients know how to retry their requests as part of our Kubernetes use, but I remain interested in shortening up this race window on the Minio S3 side of things. That should buy us time to do things "the right way".

Thanks!

[harshavardhana]

My question for this forum is: Is there a way of creating a bucket an instant after Minio S3 comes up, as part of Minio S3's bootstrap process? Our software always uses the same bucket name and creds for a given environment.

you should run mc as a job to create the bucket before your other services start.