dockerutil.BuildEmptyImage() fails with bad API version #646

jimcottrell · 2024-02-26T18:55:55Z

I've got a CI/CD pipeline that's run occasionally when we rebuild a image and create an optimized version with Slim. Without anything having changed that I'm aware of (same Slim version, same pipeline runner), this process has started failing with:

dockerutil.BuildEmptyImage: dockerapi.BuildImage() error = API error (400): client version 1.12 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version

I thought this might be related to #637, but it seems to be a separate issue, and the same thing happens whether or not I try setting --crt-api-version. Having updated everything to the latest versions, the problem continues. The issue seems to be specific to this environment, as a local build works.

Steps to Reproduce the Problem

slim build --http-probe-off --continue-after=enter alpine

(trivial reproduction; of course not our actual image)

cmd=build state=started
cmd=build info=params target.image='alpine' continue.mode='enter' rt.as.user='true' keep.perms='true' tags='' image-build-engine='internal' target.type='image' 
cmd=build state=image.inspection.start
cmd=build info=image id='sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd' size.bytes='0' size.human='0 B' 
cmd=build info=image.stack index='0' name='alpine:latest' id='sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd' 
cmd=build state=image.inspection.done
cmd=build state=container.inspection.start
time="2024-02-26T18:12:39Z" level=error msg="dockerutil.BuildEmptyImage: dockerapi.BuildImage() error = API error (400): client version 1.12 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version\n / output: "
time="2024-02-26T18:12:39Z" level=fatal msg="slim: failure" error="API error (400): client version 1.12 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version\n" stack="goroutine 1 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x5e\ngithub.com/slimtoolkit/slim/pkg/util/errutil.FailOn({0x231e3e0, 0xc0007e62a0})\n\tgithub.com/slimtoolkit/slim/pkg/util/errutil/errutil.go:32 +0x4b\ngithub.com/slimtoolkit/slim/pkg/app/master/inspectors/container.(*Inspector).RunContainer(0xc000464b00)\n\tgithub.com/slimtoolkit/slim/pkg/app/master/inspectors/container/container_inspector.go:430 +0x77a\ngithub.com/slimtoolkit/slim/pkg/app/master/command/build.OnCommand(_, _, {_, _}, _, {_, _}, {_, _}, {0x0, ...}, ...)\n\tgithub.com/slimtoolkit/slim/pkg/app/master/command/build/handler.go:1132 +0x4632\[ngithub.com/slimtoolkit/slim/pkg/app/master/command/build.glob..func1(0xc0003d35c0)\n\tgithub.com/slimtoolkit/slim/pkg/app/master/command/build/cli.go:760](http://ngithub.com/slimtoolkit/slim/pkg/app/master/command/build.glob..func1(0xc0003d35c0)/n/tgithub.com/slimtoolkit/slim/pkg/app/master/command/build/cli.go:760) +0x5d51\[ngithub.com/urfave/cli/v2.(*Command](http://ngithub.com/urfave/cli/v2.(*Command)).Run(0x3396180, 0xc0000af480)\n\[tgithub.com/urfave/cli/v2@v2.3.0/command.go:163](http://tgithub.com/urfave/cli/v2@v2.3.0/command.go:163) +0x583\[ngithub.com/urfave/cli/v2.(*App](http://ngithub.com/urfave/cli/v2.(*App)).RunContext(0xc000266d00, {0x233bb88?, 0x340d500}, {0xc00012e000, 0x7, 0x7})\n\[tgithub.com/urfave/cli/v2@v2.3.0/app.go:313](http://tgithub.com/urfave/cli/v2@v2.3.0/app.go:313) +0xaa5\[ngithub.com/urfave/cli/v2.(*App](http://ngithub.com/urfave/cli/v2.(*App)).Run(...)\n\[tgithub.com/urfave/cli/v2@v2.3.0/app.go:224\ngithub.com/slimtoolkit/slim/pkg/app/master.Run()\n\tgithub.com/slimtoolkit/slim/pkg/app/master/app.go:15](http://tgithub.com/urfave/cli/v2@v2.3.0/app.go:224/ngithub.com/slimtoolkit/slim/pkg/app/master.Run()/n/tgithub.com/slimtoolkit/slim/pkg/app/master/app.go:15) +0x45\nmain.main()\n\[tgithub.com/slimtoolkit/slim/cmd/slim/main.go:15](http://tgithub.com/slimtoolkit/slim/cmd/slim/main.go:15) +0x187\n" version="linux/amd64|Transformer|1.40.11|1b271555882eacdfb4e6598d6d0552e9b9b1449b|2024-02-02_01:36:22PM"

Specifications

Version: 1.40.11
Platform: linux/amd64 (self-hosted Bitbucket pipeline runner, which is configured for privileged Docker access so Slim can run)
Output of slim version:

cmd=version info=app container='true' dsimage='false' location='/opt/atlassian/pipelines/agent/build/dist_linux' current='1.40.11' verdict='you have the latest version' cmd='version' version='linux/amd64|Transformer|1.40.11|1b271555882eacdfb4e6598d6d0552e9b9b1449b|2024-02-02_01:36:22PM' 
cmd=version info=host cmd='version' osname='Ubuntu 22.04.3 LTS' osbuild='' version='#18~22.04.1-Ubuntu SMP Wed Jan 10 22:54:16 UTC 2024' release='6.2.0-1018-aws' sysname='Linux' 
cmd=version info=docker server.version='25.0.3' architecture='x86_64' cmd='version' name='a640edfcc04d' kernel.version='6.2.0-1018-aws' operating.system='Alpine Linux v3.19 (containerized)' ostype='linux' 
cmd=version info=dclient build.time='2024-02-06T21:13:08.000000000+00:00' git.commit='f417435' cmd='version' api.version='1.44' min.api.version='1.24' 
app='slim' message='GitHub Discussions' info='https://github.com/slimtoolkit/slim/discussions'
app='slim' message='Join the CNCF Slack channel to ask questions or to share your feedback' info='https://cloud-native.slack.com/archives/C059QP1RH1S'
app='slim' message='Join the Discord server to ask questions or to share your feedback' info='https://discord.gg/9tDyxYS'
app='slim' message='Join the Gitter channel to ask questions or to share your feedback' info='https://gitter.im/docker-slim/community'

The text was updated successfully, but these errors were encountered:

kcq · 2024-02-28T08:17:09Z

Interesting... need to investigate/repro. Either way, the next release should address this. Check back a bit later or follow me on Github or Twitter to be notified when the new release is out.

jimcottrell · 2024-02-28T17:37:31Z

Sounds good, thanks. I'll see how the next version works out. We're still having some build issues, but it looks like if we set a DOCKER_API_VERSION environment variable, we can at least get past this error.

jimcottrell · 2024-02-28T18:23:14Z

For reference, other versions may work (1.44 did not), but we seem to be back in business with

export DOCKER_API_VERSION='1.25'

in our pipeline

kcq · 2024-04-06T07:07:55Z

@jimcottrell this might work as-is https://github.com/mintoolkit/mint/releases/tag/1.41.1 let me know if you still have problems there

jimcottrell · 2024-04-26T17:19:03Z

I've tried with v1.41.1 and no explicit DOCKER_API_VERSION and I get that same error: client version 1.12 is too old

kcq · 2024-04-26T18:54:13Z

@jimcottrell which version of Docker do you have installed? Are you on Mac or Linux? If you are on Linux are you using Docker Desktop or only Docker Engine? If you are on a Mac is it an AMD or an M1/M2? Trying to repro... more info on your environment will help

sammcj · 2024-05-05T04:00:29Z

Getting this on Fedora 39 with the latest stable docker-ce and slim installed.

slim build --target nvapi --tag slim --http-probe=false --dockerfile Dockerfile --dockerfile-context .
...
cmd=build info=build.error status='standard.image.build.error' value='API /build requires version 1.25, requested version 1.24 is insufficient'
cmd=build state=exited code=33554435 version=linux/amd64|Transformer|1.40.11|1b271555882eacdfb4e6598d6d0552e9b9b1449b|2024-02-02_01:36:22PM location=/usr/local/bin

dnf info docker-ce-26.1.1-1.fc39.x86_64
Last metadata expiration check: 4:53:24 ago on Sun 05 May 2024 09:04:18.
Installed Packages
Name         : docker-ce
Epoch        : 3
Version      : 26.1.1
Release      : 1.fc39
Architecture : x86_64
Size         : 104 M
Source       : docker-ce-26.1.1-1.fc39.src.rpm
Repository   : @System
From repo    : docker-ce-stable
Summary      : The open-source application container engine
URL          : https://www.docker.com

Setting DOCKER_API_VERSION='1.25' makes no difference.

Seems to be a common problem looking at #637 and caused by #641

kcq · 2024-05-05T18:32:23Z

@sammcj do you get the same result with this version?

kcq · 2024-05-05T18:35:54Z

@sammcj Here's the Vagrant-based repro with Fedora 39 and Docker 26.1.1. You should be good with the release version in the repro config. The release version you shared in this issue is out of date.

jimcottrell · 2024-06-03T18:03:28Z

@kcq Sorry for the delay in responding. As noted before, this is happening only on a self-hosted Bitbucket pipeline runner (linux/amd64). When I run locally, I have no issue. To make Slim work in the pipeline environment, it has to run via the docker:dind image.

Host Docker

+ docker version
Client:
 Version:           20.10.24
 API version:       1.41
 Go version:        go1.19.7
 Git commit:        297e128
 Built:             Tue Apr  4 18:17:06 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          25.0.2
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       fce6e0c
  Built:            Thu Feb  1 00:23:45 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.13
  GitCommit:        7c3aca7a610df76212171d200ca3811ff6096eb8
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

`docker:dind`

+ docker version
Client:
 Version:           20.10.24
 API version:       1.41
 Go version:        go1.19.7
 Git commit:        297e128
 Built:             Tue Apr  4 18:17:06 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          26.1.3
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       8e96db1
  Built:            Thu May 16 08:33:58 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.15
  GitCommit:        926c9586fe4a6236699318391cd44976a98e31f1
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Looking at this does seem to explain why I couldn't set it to use v1.44.

kcq · 2024-06-05T00:13:05Z

@kcq Sorry for the delay in responding. As noted before, this is happening only on a self-hosted Bitbucket pipeline runner (linux/amd64). When I run locally, I have no issue. To make Slim work in the pipeline environment, it has to run via the docker:dind image.

Host Docker

+ docker version
Client:
 Version:           20.10.24
 API version:       1.41
 Go version:        go1.19.7
 Git commit:        297e128
 Built:             Tue Apr  4 18:17:06 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          25.0.2
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       fce6e0c
  Built:            Thu Feb  1 00:23:45 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.13
  GitCommit:        7c3aca7a610df76212171d200ca3811ff6096eb8
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

`docker:dind`

+ docker version
Client:
 Version:           20.10.24
 API version:       1.41
 Go version:        go1.19.7
 Git commit:        297e128
 Built:             Tue Apr  4 18:17:06 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          26.1.3
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       8e96db1
  Built:            Thu May 16 08:33:58 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.15
  GitCommit:        926c9586fe4a6236699318391cd44976a98e31f1
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Looking at this does seem to explain why I couldn't set it to use v1.44.

@jimcottrell Running with dind in Bitbucket runners makes it more complicated because each CI/CD vendor does the dind setup in slightly different ways, which requires different ways to connect to the (correct) Docker runtime. Some environments don't expose the Docker unix socket. Other environments need certs to connect to the Docker runtime. Some have more advanced network configurations that prevents direct container communication, so the --sensor-ipc-mode and --sensor-ipc-endpoint flags need to be used. The RUNNING CONTAINERIZED README section covers a few examples

To troubleshoot your setup we need two things to get started:

How you invoke the slim command in your dind setup (if it's slim build --http-probe-off --continue-after=enter alpine then it needs to be updated to point to the dind Docker runtime. optionally you can try using the containerized version of the app, but that's not required).
Your Bitbucket runner config that defines how the dind magic is configured.

jimcottrell · 2024-06-05T23:18:05Z

Here's a simplified version of our bitbucket-pipelines.yml that fails with client version 1.12 is too old:

image:
  name: atlassian/default-image:4
definitions:
  services:
    privileged-docker:
      type: docker
      image:
        name: docker:dind
pipelines:
  custom:
    test:
      - step:
          runs-on:
            - linux
            - self.hosted
          name: Build Optimized Image
          services:
            - privileged-docker
          script:
            - curl https://downloads.dockerslim.com/releases/1.40.11/dist_linux.tar.gz -o slim.tar.gz
            - tar xvf slim.tar.gz
            #- export DOCKER_API_VERSION='1.25' # Temporary, see https://github.com/slimtoolkit/slim/issues/646
            - ./dist_linux/slim build --continue-after=enter alpine

I tried it with --in-container=true which didn't make any difference. What needs to be updated to point to the dind Docker runtime? Apart from this one case, I'm not very familiar with dind. The DOCKER_HOST environment variable exists, but I don't believe the Docker UNIX socket is present.

paulcalabro mentioned this issue Apr 20, 2024

slim 1.40.9 not compatible with new release docker 25.0 #632

Closed

sammcj mentioned this issue May 5, 2024

fix: Increase min Docker API verison requirement to 1.25 #641

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dockerutil.BuildEmptyImage() fails with bad API version #646

dockerutil.BuildEmptyImage() fails with bad API version #646

jimcottrell commented Feb 26, 2024

kcq commented Feb 28, 2024

jimcottrell commented Feb 28, 2024

jimcottrell commented Feb 28, 2024

kcq commented Apr 6, 2024

jimcottrell commented Apr 26, 2024

kcq commented Apr 26, 2024

sammcj commented May 5, 2024 •

edited

kcq commented May 5, 2024

kcq commented May 5, 2024 •

edited

jimcottrell commented Jun 3, 2024

kcq commented Jun 5, 2024 •

edited

Host Docker

`docker:dind`

jimcottrell commented Jun 5, 2024

dockerutil.BuildEmptyImage() fails with bad API version #646

dockerutil.BuildEmptyImage() fails with bad API version #646

Comments

jimcottrell commented Feb 26, 2024

Steps to Reproduce the Problem

Specifications

kcq commented Feb 28, 2024

jimcottrell commented Feb 28, 2024

jimcottrell commented Feb 28, 2024

kcq commented Apr 6, 2024

jimcottrell commented Apr 26, 2024

kcq commented Apr 26, 2024

sammcj commented May 5, 2024 • edited

kcq commented May 5, 2024

kcq commented May 5, 2024 • edited

jimcottrell commented Jun 3, 2024

Host Docker

docker:dind

kcq commented Jun 5, 2024 • edited

Host Docker

docker:dind

jimcottrell commented Jun 5, 2024

sammcj commented May 5, 2024 •

edited

kcq commented May 5, 2024 •

edited

`docker:dind`

kcq commented Jun 5, 2024 •

edited

`docker:dind`