When using GCR and mount the config.json - got: No credentials for gcr.io found

[igal-antidote]

**watchtower with GCP mount to config.json not recognize the file **

To Reproduce

Steps to reproduce the behavior:

1. Setup GCP instance with Service account permissions to pull from gcr.io
2. run gcloud auth configure-docker
3. Create docker-compose with

   image: containrrr/watchtower
    restart: always
    environment:
      WATCHTOWER_NOTIFICATIONS: ${WATCHTOWER_NOTIFICATIONS}
      WATCHTOWER_NOTIFICATION_SLACK_HOOK_URL: ${WATCHTOWER_NOTIFICATION_SLACK_HOOK_URL}
      WATCHTOWER_NOTIFICATION_SLACK_IDENTIFIER: ${WATCHTOWER_NOTIFICATION_SLACK_IDENTIFIER}
      WATCHTOWER_NOTIFICATION_SLACK_CHANNEL: ${WATCHTOWER_NOTIFICATION_SLACK_CHANNEL}
      HOME: /root
      GCR: "yes"
    volumes:  
      - /var/run/docker.sock:/var/run/docker.sock
      - $HOME/.docker/config.json:/root/.docker/config.json
      - $HOME/.config/gcloud:/root/.config/gcloud
   command: --cleanup --debug --interval 15
    labels:
        - com.centurylinklabs.watchtower.enable=false

4. watch docker logs

time="2022-08-07T09:58:50Z" level=debug
time="2022-08-07T09:58:50Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2022-08-07T09:58:51Z" level=debug msg="Making sure everything is sane before starting"
time="2022-08-07T09:58:51Z" level=debug msg="Retrieving running containers"
time="2022-08-07T09:58:51Z" level=debug msg="There are no additional watchtower containers"
time="2022-08-07T09:58:51Z" level=debug msg="Watchtower HTTP API skipped."
time="2022-08-07T09:58:51Z" level=info msg="Watchtower 1.4.0"
time="2022-08-07T09:58:51Z" level=info msg="Using notifications: slack"
time="2022-08-07T09:58:51Z" level=info msg="Checking all containers (except explicitly disabled with label)"
time="2022-08-07T09:58:51Z" level=info msg="Scheduling first run: 2022-08-07 09:59:06 +0000 UTC"
time="2022-08-07T09:58:51Z" level=info msg="Note that the first check will be performed in 14 seconds"
time="2022-08-07T09:59:06Z" level=debug msg="Checking containers for updated images"
time="2022-08-07T09:59:06Z" level=debug msg="Retrieving running containers"
time="2022-08-07T09:59:06Z" level=debug msg="Trying to load authentication credentials." container=/antidote-queries-1 image="gcr.io/med-and-beyond/beyond-queries:staging-latest"
time="2022-08-07T09:59:06Z" level=debug msg="No credentials for gcr.io found" config_file=/config.json
time="2022-08-07T09:59:06Z" level=debug msg="Got image name: gcr.io/xxx-xxx-xxx/beyond-queries:staging-latest"
time="2022-08-07T09:59:06Z" level=debug msg="Checking if pull is needed" container=/image-queries-1 image="gcr.io/xxx-xxx-xxx/beyond-queries:staging-latest"
time="2022-08-07T09:59:06Z" level=debug msg="Building challenge URL" URL="https://gcr.io/v2/"
time="2022-08-07T09:59:06Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://gcr.io/v2/token\",service=\"gcr.io\"" status="401 Unauthorized"
time="2022-08-07T09:59:06Z" level=debug msg="Checking challenge header content" realm="https://gcr.io/v2/token" service=gcr.io
time="2022-08-07T09:59:06Z" level=debug msg="Setting scope for auth token" image=gcr.io/xxx-xxx-xxx/beyond-queries scope="repository:gcr.io/xxx-xxx-xxx/beyond-queries:pull"
time="2022-08-07T09:59:06Z" level=debug msg="No credentials found."
time="2022-08-07T09:59:06Z" level=debug msg="Parsing image ref" host=gcr.io image=xxx-xxx-xxx/beyond-queries normalized="gcr.io/xxx-xxx-xxx/beyond-queries:staging-latest" tag=staging-latest
time="2022-08-07T09:59:06Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://gcr.io/v2/xxx-xxx-xxx/beyond-queries/manifests/staging-latest"
time="2022-08-07T09:59:06Z" level=debug msg="Could not do a head request for \"gcr.io/xxx-xxx-xxx/beyond-queries:staging-latest\", falling back to regular pull." container=/image-queries-1 image="gcr.io/xxx-xxx-xxx/beyond-queries:staging-latest"
time="2022-08-07T09:59:06Z" level=debug msg="Reason: registry responded to head request with \"401 Unauthorized\", auth: \"Bearer realm=\\\"https://gcr.io/v2/token\\\",service=\\\"gcr.io\\\",scope=\\\"repository:xxx-xxx-xxx/beyond-queries:pull\\\"\"" container=/image-queries-1 image="gcr.io/med-and-beyond/beyond-queries:staging-latest"
time="2022-08-07T09:59:06Z" level=debug msg="Pulling image" container=/antidote-queries-1 image="gcr.io/med-and-beyond/beyond-queries:staging-latest"
time="2022-08-07T09:59:06Z" level=debug msg="Error pulling image gcr.io/xxx-xxx-xxx/beyond-queries:staging-latest, Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication"
time="2022-08-07T09:59:06Z" level=info msg="Unable to update container \"/image-queries-1\": Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication. Proceeding to next."

Expected behavior
should be pull and restart as expected in watchtower documentation

Screenshots
Do not have special screenshots

Environment
We want to know:

• Google Cloud Platform
• Goolge instance virtual machine, ubuntu 20.04
• Docker version:

Version:           20.10.17
API version:       1.41
Go version:        go1.17.11
Git commit:        100c701
Built:             Mon Jun  6 23:02:57 2022
OS/Arch:           linux/amd64
Context:           default
Experimental:      true

Server: Docker Engine - Community
Engine:
 Version:          20.10.17
 API version:      1.41 (minimum version 1.12)
 Go version:       go1.17.11
 Git commit:       a89b842
 Built:            Mon Jun  6 23:01:03 2022
 OS/Arch:          linux/amd64
 Experimental:     false
containerd:
 Version:          1.6.6
 GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
runc:
 Version:          1.1.2
 GitCommit:        v1.1.2-0-ga916309
docker-init:
 Version:          0.19.0
 GitCommit:        de40ad0```

What am i missing here?

[piksel]

Watchtower reports that it does not find an entry for gcr.io:

time="2022-08-07T09:59:06Z" level=debug msg="No credentials for gcr.io found" config_file=/config.json

Does your config.json contain any such entry?
What does

Setup GCP instance with Service account permissions to pull from gcr.io

mean specifically?

Also, the config file should be mounted as /config.json as per the docs.
You have it mounted in /root/.docker/config.json.

[igal-antidote]

After an investigation, i found that watchtower is not working well with creds helper in GCP.

creds helper is done by this commnad:

gcloud auth configure-docker

And we assume that the VM have Service account with permissions to access GCR.

so what i did is to rebuild the container with google-sdk and docker, and created entrypoint that runs before /watchtower the following command

docker login -u oauth2accesstoken -p "$(gcloud auth print-access-token)" https://gcr.io

then i copy the inside in the entrypoint the config.json into /config.json
and it works well.

conclusion: gcr credential helpers does not work well for now and by my best tests

the entrypoint.sh:

#!/bin/bash
set -e

if [ "$GCR" == "yes" ]  ; then 
   docker login -u oauth2accesstoken -p "$(gcloud auth print-access-token)" https://gcr.io
   if [ -f /root/.docker/config.json ] ; then
	   cp /root/.docker/config.json /config.json
   fi
fi

exec /watchtower "$@"

[simskij]

That sounds odd. Did you try setting the credential helper up with the help of a separate container and a shared volume, as described in the docs?

[Itsindigo]

This isn't ideal, as the access token's generated by gcloud auth print-access-token are only shortlived

[piksel]

You should be able to use

docker login -u _json_key -p "$(cat gcloudauth.json)" https://gcr.io

to add the credentials for logging in using exported json credentials.

[Sankalan13]

I was hoping to use watchtower for our local end to end docker compose testing setup. It'd be a life saver and so very helpful for testers. But we are using GCP cred helpers as well and I faced the same issue with passing in the token to watchtower. I don't understand the solution described here very well. This might be a bit stale but could you help me understand how you got it working using the entrypoint.sh file?

[Sankalan13]

This is a bug that is blocking me from using watchtower as well. Please help with a solution that is more straightforward.