[BUG] docker-compose build is not caching layers

[ldanielw1]

Description

I'm on an M1 mac with colima. When I try to run docker build . in my repo, it caches layers - but when I run docker-compose build, it doesn't. AND it deletes the layers cached when I had run docker build .
I also noticed that every time I run docker-compose build, the number of layers is increasing in my colima machine's /var/lib/docker/overlay2. So I'm guessing it just can't find the cached layers and is continuing to create new layers in that folder?

I'm curious why this might be the case and how I can get my docker-compose builds to start caching layers again?

Steps To Reproduce

No response

Compose Version

Docker Compose version 2.27.0
Docker Compose version 2.27.0

Docker Environment

Client: Docker Engine - Community
 Version:    26.1.1
 Context:    colima
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc.)
    Version:  2.27.0
    Path:     /Users/dwu/.docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 16
 Server Version: 24.0.9
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.5.0-15-generic
 Operating System: Ubuntu 23.10
 OSType: linux
 Architecture: aarch64
 CPUs: 2
 Total Memory: 1.851GiB
 Name: colima
 ID: b35b721d-3835-4a57-b470-3205fbb35d23
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[ndeloof]

compose does not run the build on it's own but just creates a build configuration for buildkit to build image(s)
Can you reproduce this behaviour with Docker Desktop? Otherwise this is a Colima issue you should report to the authors

[ldanielw1]

that's the crazy thing - I can't reproduce this behavior with Docker Desktop. But I also can't reproduce this issue with docker on colima. So something is different between how docker-compose is creating layers, vs. how docker is doing it. So - wouldn't that be an issue of how docker-compose differs from docker? I could be naive, though, so lmk if there's something I'm missing (I have already raised this issue with them, but am still waiting for a response)

[ndeloof]

that's indeed probably an issue with compose configuring build execution (compose just setup buildx to run the build, it doesn't create layers on its own). We will welcome any PR to offer a fix so this works well on Colima, but you will easily understand that, as a Docker employee, I can't spend time debugging an issue that only applies to a competing product :)

[neelp03]

Have you tried modifying the BuildKit settings directly if possible, or forcing Docker Compose to use a different build method or configuration to see if that affects the caching behavior?

Also could you compare the build context from Docker vs. Docker Compose? Sometimes there might be differences in how the context is packaged and sent to the builder which can affect caching.

[ldanielw1]

@ndeloof - of course, I appreciate your pointing me in the right direction, though.
@neelp03 - well, i'm currently running builds with BuildKit turned off (with a DOCKER_BUILDKIT=0). And strangely enough, now that you're asking, the problem goes away if I set it back to 1. However, I kind of need buildkit to be off for the time-being (for debug purposes).

It guess there is something about how compose runs when on colima AND when buildkit is off, where it is creating layers uniquely from build to build such that it always has a cache-miss before building layers...