What happened?

TASK [download : Download_container | Download image if required] ****************************************************************************************************************************************************
fatal: [worker-2]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/local/bin/ctr", "-n", "k8s.io", "images", "pull", "rancher/local-path-provisioner:v0.0.19"], "delta": "0:00:00.055235", "end": "2024-04-30 22:05:49.405116", "msg": "non-zero return code", "rc": 1, "start": "2024-04-30 22:05:49.349881", "stderr": "time="2024-04-30T22:05:49Z" level=info msg="trying next host" error="failed to do request: Head \"https://rancher/v2/local-path-provisioner/manifests/v0.0.19\\\": dial tcp: lookup rancher on 8.8.8.8:53: no such host" host=rancher\nctr: failed to resolve reference "rancher/local-path-provisioner:v0.0.19": failed to do request: Head "https://rancher/v2/local-path-provisioner/manifests/v0.0.19\": dial tcp: lookup rancher on 8.8.8.8:53: no such host", "stderr_lines": ["time="2024-04-30T22:05:49Z" level=info msg="trying next host" error="failed to do request: Head \"https://rancher/v2/local-path-provisioner/manifests/v0.0.19\": dial tcp: lookup rancher on 8.8.8.8:53: no such host" host=rancher", "ctr: failed to resolve reference "rancher/local-path-provisioner:v0.0.19": failed to do request: Head "https://rancher/v2/local-path-provisioner/manifests/v0.0.19": dial tcp: lookup rancher on 8.8.8.8:53: no such host"], "stdout": "", "stdout_lines": []}

What did you expect to happen?

local_path_provisioner_enabled: true
local_path_provisioner_namespace: "local-path-storage"
local_path_provisioner_storage_class: "local-path"
local_path_provisioner_reclaim_policy: Delete
local_path_provisioner_claim_root: /opt/local-path-provisioner/
local_path_provisioner_debug: false
local_path_provisioner_image_repo: "rancher/local-path-provisioner"
local_path_provisioner_image_tag: "v0.0.19"
local_path_provisioner_helper_image_repo: "busybox"
local_path_provisioner_helper_image_tag: "latest"

How can we reproduce it (as minimally and precisely as possible)?

cat k8s-cluster.yml

kube_config_dir: /etc/kubernetes
kube_script_dir: "{{ bin_dir }}/kubernetes-scripts"
kube_manifest_dir: "{{ kube_config_dir }}/manifests"
kube_cert_dir: "{{ kube_config_dir }}/ssl"
kube_token_dir: "{{ kube_config_dir }}/tokens"
kube_api_anonymous_auth: true
kube_version: v1.29.3
local_release_dir: "/tmp/releases"
retry_stagger: 5
kube_cert_group: kube-cert
kube_log_level: 2
credentials_dir: "{{ inventory_dir }}/credentials"
kube_network_plugin: calico
kube_network_plugin_multus: false
kube_service_addresses: 10.233.0.0/18
kube_pods_subnet: 10.233.64.0/18
kube_network_node_prefix: 24
calico_pool_blocksize: 24
enable_dual_stack_networks: false
kube_apiserver_ip: "{{ kube_service_addresses|ipaddr('net')|ipaddr(1)|ipaddr('address') }}"
kube_apiserver_port: 6443 # (https)
kube_apiserver_insecure_port: 0 # (disabled)
kube_proxy_mode: iptables
kube_proxy_strict_arp: false
kube_proxy_nodeport_addresses: >-
{%- if kube_proxy_nodeport_addresses_cidr is defined -%}
[{{ kube_proxy_nodeport_addresses_cidr }}]
{%- else -%}
[]
{%- endif -%}

kube_encrypt_secret_data: false
cluster_name: cluster.local
ndots: 2
dns_mode: coredns
enable_nodelocaldns: true
enable_nodelocaldns_secondary: false
nodelocaldns_ip: 10.233.0.4
nodelocaldns_health_port: 9254
nodelocaldns_second_health_port: 9256
nodelocaldns_bind_metrics_host_ip: false
nodelocaldns_secondary_skew_seconds: 5
enable_coredns_k8s_external: false
coredns_k8s_external_zone: k8s_external.local
enable_coredns_k8s_endpoint_pod_names: false
resolvconf_mode: host_resolvconf
deploy_netchecker: false
skydns_server: "{{ kube_service_addresses|ipaddr('net')|ipaddr(3)|ipaddr('address') }}"
skydns_server_secondary: "{{ kube_service_addresses|ipaddr('net')|ipaddr(4)|ipaddr('address') }}"
dns_domain: "{{ cluster_name }}"

kata_containers_enabled: false
kubeadm_certificate_key: "{{ lookup('password', credentials_dir + '/kubeadm_certificate_key.creds length=64 chars=hexdigits') | lower }}"
k8s_image_pull_policy: IfNotPresent
kubernetes_audit: false
dynamic_kubelet_configuration: false
default_kubelet_config_dir: "{{ kube_config_dir }}/dynamic_kubelet_dir"
dynamic_kubelet_configuration_dir: "{{ kubelet_config_dir | default(default_kubelet_config_dir) }}"
podsecuritypolicy_enabled: false

Amount of time to retain events. (default 1h0m0s)

event_ttl_duration: "1h0m0s"

Automatically renew K8S control plane certificates on first Monday of each month

auto_renew_certificates: false

First Monday of each month

auto_renew_certificates_systemd_calendar: "Mon --1,2,3,4,5,6,7 03:{{ groups['kube_control_plane'].index(inventory_hostname) }}0:00"

local_path_provisioner_enabled: true
local_path_provisioner_namespace: "local-path-storage"
local_path_provisioner_storage_class: "local-path"
local_path_provisioner_reclaim_policy: Delete
local_path_provisioner_claim_root: /opt/local-path-provisioner/
local_path_provisioner_debug: false
local_path_provisioner_image_repo: "rancher/local-path-provisioner"
local_path_provisioner_image_tag: "v0.0.24"
local_path_provisioner_helper_image_repo: "busybox"
local_path_provisioner_helper_image_tag: "latest"

local_volume_provisioner_enabled: true
local_volume_provisioner_namespace: kube-system
local_volume_provisioner_nodelabels:

• kubernetes.io/hostname
  local_volume_provisioner_storage_classes:
  local-storage:
  host_dir: /mnt/data
  mount_dir: /mnt/data
  volume_mode: Filesystem
  fs_type: ext4

OS

ubuntu22

Version of Ansible

2.16.6

Version of Python

3.12

Version of Kubespray (commit)

main and v2.24.1

Network plugin used

calico

Full inventory with variables

ansible-playbook upgrade-cluster.yml -b -i inventory/files/inventory.ini -e kube_version=v1.29.3 --limit "master-1:worker-2:worker-3"

Command used to invoke ansible

none

Output of ansible run

TASK [download : Download_container | Download image if required] ****************************************************************************************************************************************************
fatal: [worker-2]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/local/bin/ctr", "-n", "k8s.io", "images", "pull", "rancher/local-path-provisioner:v0.0.19"], "delta": "0:00:00.055235", "end": "2024-04-30 22:05:49.405116", "msg": "non-zero return code", "rc": 1, "start": "2024-04-30 22:05:49.349881", "stderr": "time="2024-04-30T22:05:49Z" level=info msg="trying next host" error="failed to do request: Head \"https://rancher/v2/local-path-provisioner/manifests/v0.0.19\\\": dial tcp: lookup rancher on 8.8.8.8:53: no such host" host=rancher\nctr: failed to resolve reference "rancher/local-path-provisioner:v0.0.19": failed to do request: Head "https://rancher/v2/local-path-provisioner/manifests/v0.0.19\": dial tcp: lookup rancher on 8.8.8.8:53: no such host", "stderr_lines": ["time="2024-04-30T22:05:49Z" level=info msg="trying next host" error="failed to do request: Head \"https://rancher/v2/local-path-provisioner/manifests/v0.0.19\": dial tcp: lookup rancher on 8.8.8.8:53: no such host" host=rancher", "ctr: failed to resolve reference "rancher/local-path-provisioner:v0.0.19": failed to do request: Head "https://rancher/v2/local-path-provisioner/manifests/v0.0.19": dial tcp: lookup rancher on 8.8.8.8:53: no such host"], "stdout": "", "stdout_lines": []}

Anything else we need to know

No response