m1: act fails to pull with unauthorized: incorrect username or password

[ssbarnea]

I am on MacOS M1 and act failed to even pull the container reporting not being able to login to docker registry, but docker works correctly and it is able to pull.

$ act
WARN  ⚠ You are using Apple M1 chip and you have not specified container architecture, you might encounter issues while running act. If so, try running it with '--container-architecture linux/amd64'. ⚠
ERRO[0000] Unable to interpolate expression 'format('{0}', matrix.env.TOXENV)': Unable to dereference 'toxenv' on non-struct 'invalid'
ERRO[0000] 'runs-on' key not defined in push/ack
ERRO[0000] 'runs-on' key not defined in push/ack
[tox/lint-1     ] 🧪  Matrix: map[env:map[TOXENV:lint] os:ubuntu-20.04 python-version:3.9]
[tox/py-3       ] 🧪  Matrix: map[env:map[TOXENV:py] os:ubuntu-20.04 python-version:3.9]
[tox/packaging-2] 🧪  Matrix: map[env:map[TOXENV:packaging] os:ubuntu-20.04 python-version:3.9]
[tox/py-3       ] 🚀  Start image=node:16-buster-slim
[tox/packaging-2] 🚀  Start image=node:16-buster-slim
[tox/lint-1     ] 🚀  Start image=node:16-buster-slim
[tox/py-3       ]   🐳  docker pull image=node:16-buster-slim platform= username= forcePull=false
[tox/packaging-2]   🐳  docker pull image=node:16-buster-slim platform= username= forcePull=false
[tox/lint-1     ]   🐳  docker pull image=node:16-buster-slim platform= username= forcePull=false
Error: Error response from daemon: Head "https://registry-1.docker.io/v2/library/node/manifests/16-buster-slim": unauthorized: incorrect username or password
FAIL: 1

ssbarnea@m1: ~/c/schemastorepy main
$ docker pull node:16-buster-slim
16-buster-slim: Pulling from library/node
383d573fdcf2: Pull complete
a3942ef45b19: Pull complete
233507218b05: Pull complete
d14f918387bf: Pull complete
9f12e5ac2608: Pull complete
Digest: sha256:e5936cf8dbbb3c0d9d337c0e9f23e56e00b7e62280a8a617d12adfdda8ca9f11
Status: Downloaded newer image for node:16-buster-slim
docker.io/library/node:16-buster-slim

ssbarnea@m1: ~/c/schemastorepy main
$ docker --version
Docker version 20.10.14, build a224086

[catthehacker]

Can you show your workflow and secrets/env?
We use in-workflow credentials, credentials passed into act and docker cli/desktop credential helpers.

act/pkg/runner/run_context.go

Lines 657 to 658 in 91fd412

	username = rc.Config.Secrets["DOCKER_USERNAME"]
	password = rc.Config.Secrets["DOCKER_PASSWORD"]

act/pkg/runner/run_context.go

Line 680 in 91fd412

if container.Credentials["username"] == "" || container.Credentials["password"] == "" {

https://github.com/nektos/act/blob/b910a42edfab7a02b08a52ecef203fd419725642/pkg/container/docker_auth.go

[ZauberNerd]

@nikhilbadyal and what's the content (make sure to redact confidential information) of your ~/.docker/config.json?

Also, just as an experiment: Could you try to rename/move the docker config and try to run act again? Public docker hub usually should not require authentication, but we are using credentials, if they are configured.
And as written here: #1165 (comment) it looks like act might select the wrong credentials. So by trying it out without a docker config, we could verify, that the issue is indeed related to the authentication code.

[nikhilbadyal]

cat ~/.docker/config.json

{
	"auths": {
		"https://index.docker.io/v1/": {}
	},
	"credsStore": "desktop"
}

[nikhilbadyal]

Even after removing config.json to fgerg.json

➜  test act -v
DEBU[0000] Loading environment from ..../test/.env
DEBU[0000] Loading secrets from ..../test/.secrets
DEBU[0000] Loading workflows from '....test/.github/workflows'
DEBU[0000] Loading workflows recursively
DEBU[0000] Found workflow 'ci.yml' in '..../test/.github/workflows/ci.yml'
DEBU[0000] Reading workflow '.../test/.github/workflows/ci.yml'
DEBU[0000] Planning event: push
WARN[0000] unable to get git repo: unable to find git repo
WARN[0000] unable to get git ref: unable to find git repo
WARN[0000] unable to get git revision: unable to find git repo
DEBU[0000] evaluating expression 'success()'
DEBU[0000] expression 'success()' evaluated to 'true'
[CI/test] 🚀  Start image=node:12-buster-slim
WARN[0000] unable to get git repo: unable to find git repo
WARN[0000] unable to get git ref: unable to find git repo
WARN[0000] unable to get git revision: unable to find git repo
[CI/test]   🐳  docker pull image=node:12-buster-slim platform=linux/amd64 username= forcePull=false
[CI/test]   🐳  docker pull node:12-buster-slim
DEBU[0000] Image exists? false
[CI/test] pulling image 'docker.io/library/node:12-buster-slim' (linux/amd64)
WARN[0003] unable to get git repo: unable to find git repo
WARN[0003] unable to get git ref: unable to find git repo
WARN[0003] unable to get git revision: unable to find git repo
Error: Error response from daemon: Head "https://registry-1.docker.io/v2/library/node/manifests/12-buster-slim": unauthorized: please use personal access token to login

[ZauberNerd]

Hm. That's strange...
If you have golang installed, you could clone this repository: https://github.com/ZauberNerd/docker-pull-test and execute go run main.go (remove the node:12 image before by running docker rmi node:12-buster-slim).

[nikhilbadyal]

This is working. I was able to clone

[ZauberNerd]

Hmm. At a first glance I don't see why this should fail.

It could be that our auth code is a bit too naive: https://github.com/nektos/act/blob/master/pkg/container/docker_auth.go#L29

In the docker CLI there's a comment about legacy hostnames: https://github.com/docker/cli/blob/master/vendor/github.com/docker/docker/registry/config.go#L25-L47
maybe we need to expand our auth code a bit more.

I'd be interested to know what the values of hostname and authConfig are here: https://github.com/nektos/act/blob/master/pkg/container/docker_auth.go#L29

[ssbarnea]

Based on on my recent experience, I would say that GHA has zero support for arm64 architecture. This means that it makes almost no sense not to automatically add --container-architecture linux/amd64 switch instead of asking act user to add it.

Second issue is that now I am looking at a static line about pulling and I have no idea if the pull is really happening or not as there is zero feedback on the console. Looking a CPU load does not give me much feedback either. Based on Activity Monitor I do see a little bit of incoming traffic from com.docker.vpnkit process, so I can guess that probably it is downloading. still, no idea if it will finish before I type this, in an hour or never. CLI docker does display more info, so I am inclined to find the use of API for pulling as a notable downgrade.

I guess [check/check] 🐳 docker pull image=ghcr.io/catthehacker/ubuntu:full-20.04 platform= username= forcePull=false image could be big...

I will be back with more info.

[ZauberNerd]

Running act with -v gives more status about pulling.

[catthehacker]

I guess [check/check] 🐳 docker pull image=ghcr.io/catthehacker/ubuntu:full-20.04 platform= username= forcePull=false image could be big...

It's about 40GB, and 20GB when compressed

[cinder92]

i found a work around, run following commands

docker login --username your_username_here
docker pull node:16-buster-slim
act

that worked for me

[mloskot]

I've just installed act 0.2.43 and trying the test run according to the README.md using the github-actions-demo and it's failing with this:

$ cd D:\\github-actions-demo
$ act
[CI/test] 🚀  Start image=catthehacker/ubuntu:act-latest
[CI/test]   🐳  docker pull image=catthehacker/ubuntu:act-latest platform= username= forcePull=true
Error: Error response from daemon: Head "https://registry-1.docker.io/v2/catthehacker/ubuntu/manifests/act-latest": unauthorized: incorrect username or password

I'm reading the docs and I can not find any authentication prerequisites for the demo.

Is there any secret required?

[catthehacker]

There isn't, but we do not manage the registries that host images so it could be rate-limiting on their side

[mloskot]

Right, I didn't thought this could be hitting the limits. Thank you @catthehacker for the answer

[EmmanuelTramoy]

Hi, similar issue working with my own workflow, on a Windows 10 machine :

[Is Class checkbox checked ?/check-checkbox] [DEBUG] evaluating expression 'success()'
[Is Class checkbox checked ?/check-checkbox] [DEBUG] expression 'success()' evaluated to 'true'
[Is Class checkbox checked ?/check-checkbox] 🚀  Start image=catthehacker/ubuntu:act-latest
[Is Class checkbox checked ?/check-checkbox]   🐳  docker pull image=catthehacker/ubuntu:act-latest platform= username= forcePull=true
[Is Class checkbox checked ?/check-checkbox] [DEBUG]   🐳  docker pull catthehacker/ubuntu:act-latest
[Is Class checkbox checked ?/check-checkbox] [DEBUG] pulling image 'docker.io/catthehacker/ubuntu:act-latest' ()
time="2023-03-16T10:26:25+01:00" level=debug msg="Saving notices etag=612941e9-ab25-4fd7-83eb-1d15267c66b1"
time="2023-03-16T10:26:25+01:00" level=debug msg="No new notices"
Error: Error response from daemon: Head "https://registry-1.docker.io/v2/catthehacker/ubuntu/manifests/act-latest": unauthorized: incorrect username or password

I can login with docker login and pull the docker image (says it's up to date). Tried to refresh the credentials in credential manager, no changes.

[ChristopherHX]

I saw this problem has no solution for a long time, while I never had problems myself.

Can you try this change #1774?

It logs now if it uses DockerAuthConfig credentials and if it fails pulling while authenticated it removes the credentials and tries again. However this still means you should remove stale docker auth data whereever it is located.

[lion24]

Some issue as other from a WSL2 ubuntu distro on a Windows 11 machine.

I confirm this is working fine with your patch in my setup.

Kr.

[thirtytwobits]

I'm seeing this as well on a mac (intel).

[sombriks]

removing .docker/config.json solved my issue, i had docker-desktop installed on this machine for a while but i removed that because it was using a different context. the file was dangling here afer i removed it.

[jekru]

Remove credsStore from config.json, then use docker login. Instead of docker login, you can also manually add this to the config:

"auths": {
   "https://index.docker.io/v1/": {
      "auth": ""
   }
}

The value of auth equals "username:password" but encoded in base64.