healthcheck times out

[Arelius-D]

So I'm unable to get healthy status.

Here is perhaps some useful information:
Docker version 26.1.2, build 211e74b
Docker Compose version v2.27.0

The output from docker ps

NAMES             STATE     STATUS                      NETWORKS
whats-up-docker   running   Up 9 minutes                whats-up-docker_default
ddns-updater      running   Up 26 minutes (unhealthy)   bridge
qbittorrent       running   Up 2 hours (healthy)
gluetun           running   Up 3 hours (healthy)        gluetun_default
watchtower        running   Up 16 minutes (healthy)     watchtower_default
pihole            running   Up 3 hours (healthy)        pi-hole_default

Here is some log:

========================================
========================================
============= ddns-updater =============
========================================
=========== Made with ❤️ by ============
======= https://github.com/qdm12 =======
========================================
========================================
Running version latest built on 2024-05-06T11:16:06.246Z (commit 542e895)
🔧 Need help? https://github.com/qdm12/ddns-updater/discussions/new
🐛 Bug? https://github.com/qdm12/ddns-updater/issues/new
✨ New feature? https://github.com/qdm12/ddns-updater/issues/new
☕ Discussion? https://github.com/qdm12/ddns-updater/discussions/new
💻 Email? quentin.mcgaw@gmail.com
💰 Help me? https://www.paypal.me/qmcgaw https://github.com/sponsors/qdm12
2024-05-09T21:21:04Z INFO Settings summary:
├── HTTP client
|   └── Timeout: 15s
├── Update
|   ├── Period: 5m0s
|   └── Cooldown: 5m0s
├── Public IP fetching
|   ├── HTTP enabled: yes
|   ├── HTTP IP providers
|   |   └── all
|   ├── HTTP IPv4 providers
|   |   └── all
|   ├── HTTP IPv6 providers
|   |   └── all
|   ├── DNS enabled: yes
|   ├── DNS timeout: 5s
|   └── DNS over TLS providers
|       └── all
├── Resolver: use Go default resolver
├── Server
|   ├── Listening address: :8000
|   └── Root URL: /
├── Health
|   └── Server listening address: 127.0.0.1:9999
├── Paths
|   └── Data directory: /updater/data
├── Backup: disabled
└── Logger
    ├── Level: info
    └── Caller: hidden
2024-05-09T21:21:04Z INFO reading JSON config from file /updater/data/config.json
2024-05-09T21:21:04Z INFO Found single setting to update record
2024-05-09T21:21:09Z INFO Reading history from database: domain ddns.net host assarelius ipv4
2024-05-09T21:21:09Z INFO [healthcheck server] listening on 127.0.0.1:9999
2024-05-09T21:21:09Z INFO [backup] disabled
2024-05-09T21:21:09Z INFO [http server] listening on :8000
2024-05-09T21:21:14Z INFO obtaining ipv4 address succeeded after 1 failed try
2024-05-09T21:26:14Z INFO obtaining ipv4 address succeeded after 1 failed try
2024-05-09T21:31:14Z INFO obtaining ipv4 address succeeded after 1 failed try
2024-05-09T21:36:14Z INFO obtaining ipv4 address succeeded after 1 failed try
2024-05-09T21:41:14Z INFO obtaining ipv4 address succeeded after 1 failed try
2024-05-09T21:46:14Z INFO obtaining ipv4 address succeeded after 1 failed try

and here is the output of docker inspect ... command

[
    {
        "Id": "077ae567e9e865a5d76d2ad10b30775a887c61442f1e925fca1164464ddadb08",
        "Created": "2024-05-09T21:21:03.760443253Z",
        "Path": "/updater/app",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 65827,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2024-05-09T21:21:04.080382189Z",
            "FinishedAt": "0001-01-01T00:00:00Z",
            "Health": {
                "Status": "unhealthy",
                "FailingStreak": 29,
                "Log": [
                    {
                        "Start": "2024-05-09T23:48:05.600604953+02:00",
                        "End": "2024-05-09T23:48:10.654621064+02:00",
                        "ExitCode": -1,
                        "Output": "Health check exceeded timeout (5s)"
                    },
                    {
                        "Start": "2024-05-09T23:49:10.655975348+02:00",
                        "End": "2024-05-09T23:49:15.730332965+02:00",
                        "ExitCode": -1,
                        "Output": "Health check exceeded timeout (5s)"
                    },
                    {
                        "Start": "2024-05-09T23:50:15.731673216+02:00",
                        "End": "2024-05-09T23:50:20.787870249+02:00",
                        "ExitCode": -1,
                        "Output": "Health check exceeded timeout (5s)"
                    },
                    {
                        "Start": "2024-05-09T23:51:20.789227827+02:00",
                        "End": "2024-05-09T23:51:25.850498236+02:00",
                        "ExitCode": -1,
                        "Output": "Health check exceeded timeout (5s)"
                    },
                    {
                        "Start": "2024-05-09T23:52:25.851632117+02:00",
                        "End": "2024-05-09T23:52:30.904071179+02:00",
                        "ExitCode": -1,
                        "Output": "Health check exceeded timeout (5s)"
                    }
                ]
            }
        },
        "Image": "sha256:056010abbb0f8f2533e47cb1ef4bea399149a1e7f09ca5af5adfd367f37e666c",
        "ResolvConfPath": "/var/lib/docker/containers/077ae567e9e865a5d76d2ad10b30775a887c61442f1e925fca1164464ddadb08/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/077ae567e9e865a5d76d2ad10b30775a887c61442f1e925fca1164464ddadb08/hostname",
        "HostsPath": "/var/lib/docker/containers/077ae567e9e865a5d76d2ad10b30775a887c61442f1e925fca1164464ddadb08/hosts",
        "LogPath": "/var/lib/docker/containers/077ae567e9e865a5d76d2ad10b30775a887c61442f1e925fca1164464ddadb08/077ae567e9e865a5d76d2ad10b30775a887c61442f1e925fca1164464ddadb08-json.log",
        "Name": "/ddns-updater",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/home/pi/docker/ddns-updater/data:/updater/data:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "bridge",
            "PortBindings": {
                "8000/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "8000"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": [],
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware",
                "/sys/devices/virtual/powercap"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/e88c35483f17d3d496f5a8b18a63b718fa354aff5cdb8118998570899174d982-init/diff:/var/lib/docker/overlay2/8da2ede1ac47574a0b1829c518905906d3e50038e5b355eba3f2618bad4f8dc8/diff:/var/lib/docker/overlay2/284cc582a8a4c2177d49b6647f101b39766979bcae514dab4ef3365d02acff9a/diff",
                "MergedDir": "/var/lib/docker/overlay2/e88c35483f17d3d496f5a8b18a63b718fa354aff5cdb8118998570899174d982/merged",
                "UpperDir": "/var/lib/docker/overlay2/e88c35483f17d3d496f5a8b18a63b718fa354aff5cdb8118998570899174d982/diff",
                "WorkDir": "/var/lib/docker/overlay2/e88c35483f17d3d496f5a8b18a63b718fa354aff5cdb8118998570899174d982/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/home/pi/docker/ddns-updater/data",
                "Destination": "/updater/data",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "077ae567e9e8",
            "Domainname": "",
            "User": "1000:1000",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "8000/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "UPDATE_COOLDOWN_PERIOD=5m",
                "PUBLICIP_FETCHERS=all",
                "BACKUP_PERIOD=0",
                "PUBLICIPV4_HTTP_PROVIDERS=all",
                "LISTENING_ADDRESS=:8000",
                "SHOUTRRR_ADDRESSES=",
                "PUBLICIP_DNS_PROVIDERS=all",
                "PUBLICIPV6_HTTP_PROVIDERS=all",
                "PERIOD=5m",
                "ROOT_URL=/",
                "PUBLICIP_HTTP_PROVIDERS=all",
                "LOG_CALLER=hidden",
                "LOG_LEVEL=info",
                "PUBLICIP_DNS_TIMEOUT=5s",
                "BACKUP_DIRECTORY=/updater/data",
                "CONFIG=",
                "HTTP_TIMEOUT=15s",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "DATADIR=/updater/data",
                "RESOLVER_ADDRESS=",
                "RESOLVER_TIMEOUT=5s",
                "SHOUTRRR_DEFAULT_TITLE=DDNS Updater",
                "TZ=",
                "HEALTH_SERVER_ADDRESS=127.0.0.1:9999",
                "HEALTH_HEALTHCHECKSIO_BASE_URL=https://hc-ping.com",
                "HEALTH_HEALTHCHECKSIO_UUID="
            ],
            "Cmd": null,
            "Healthcheck": {
                "Test": [
                    "CMD",
                    "/updater/app",
                    "healthcheck"
                ],
                "Interval": 60000000000,
                "Timeout": 5000000000,
                "StartPeriod": 10000000000,
                "Retries": 2
            },
            "Image": "qmcgaw/ddns-updater:latest",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": [
                "/updater/app"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "ff35c8c5901f87a612a868a270294bf28523e643ed3cb089bbdcb6061cf83165",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.image": "sha256:056010abbb0f8f2533e47cb1ef4bea399149a1e7f09ca5af5adfd367f37e666c",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "ddns-updater",
                "com.docker.compose.project.config_files": "/home/pi/docker/ddns-updater/docker-compose.yml",
                "com.docker.compose.project.working_dir": "/home/pi/docker/ddns-updater",
                "com.docker.compose.service": "ddns-updater",
                "com.docker.compose.version": "2.27.0",
                "org.opencontainers.image.authors": "quentin.mcgaw@gmail.com",
                "org.opencontainers.image.created": "2024-05-06T11:16:06.246Z",
                "org.opencontainers.image.description": "Container to update DNS records periodically with WebUI for many DNS providers",
                "org.opencontainers.image.documentation": "https://github.com/qdm12/ddns-updater",
                "org.opencontainers.image.licenses": "MIT",
                "org.opencontainers.image.revision": "542e89536c524cdb2215a48384af7ef6fad2c443",
                "org.opencontainers.image.source": "https://github.com/qdm12/ddns-updater",
                "org.opencontainers.image.title": "ddns-updater",
                "org.opencontainers.image.url": "https://github.com/qdm12/ddns-updater",
                "org.opencontainers.image.version": "latest"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "7c334566c9cb8acd1cd2f4fc94cccb6caaf55e3591a4232ed3eab722b98dfded",
            "SandboxKey": "/var/run/docker/netns/7c334566c9cb",
            "Ports": {
                "8000/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "8000"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "8000"
                    }
                ]
            },
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "f723daf63fa92cd18c4b18f7beaeea3f17fd666c0e81a8c14ced23fb8e82f92d",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "MacAddress": "02:42:ac:11:00:02",
                    "NetworkID": "b2c163e379b1bf04953c2285c4089d383d146c405561eec1e2388255814fd4d1",
                    "EndpointID": "f723daf63fa92cd18c4b18f7beaeea3f17fd666c0e81a8c14ced23fb8e82f92d",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "DriverOpts": null,
                    "DNSNames": null
                }
            }
        }
    }
]

[qdm12]

The error Health check exceeded timeout (5s) doesn't come from the ddns-updater program, but from docker. It seems it can't even launch the healthcheck program instance. The healthcheck launches an ephemeral process of ddns-updater running as a healthcheck http client querying the long running process of ddns-updater via its http healthcheck server. You could try docker exec ddns-updater /updater/app healthcheck and see if this works (check exit code with $?)? If this works, it could be a Docker bug, try upgrading it maybe?

[Arelius-D]

Btw, what do you really mean by first try? After a complete reboot? I can try it out if that is what you mean.

[qdm12]

Yes after a reboot of the container. It's strange it took 13 seconds on your logs. I'll check the code, maybe there is something holding it and that could explain the 5s timeout.

[Arelius-D]

Oh sorry for the delay but the result is I don't know what is going on, its like an endless error matrix.

So I restarted using docker restart command and whether I I waited few seconds before running the command or waiting 5 minutes the outcome was the same.

Since its not possible to post all that I turned the output into a file and have now attached it for you.

Ps, the first few times I ran it and try to log it, the file got massive, above 65MB and the limit for upload here is 25MB so.. anyhow tell me if you want me to run again for a few seconds longer. [log.txt]

[qdm12]

The log size is big due to the 'loop error' listen tcp :8000: bind: address already in use Any idea why that would happen in the container 🤔 Usually you have some other server listening on port 8000 on the same host (the container)

Another thing you can try is set the healthcheck in your docker-compose.yml with

healthcheck:
  test: ["CMD", "/updater/app", "healthcheck"]
  interval: 60s
  timeout: 15s
  start-period: 10s
  retries: 2

So the timeout is now 15s instead of 5s, does that help?

[Arelius-D]

Oh silly me, ofc.. I'm running another great container...😉 which is using that port!

And guess what, by solving another issue I was having details here I indirectly solved the problem I was having here.

So all is fine and dandy now, thank you!