Istio Multi cluster setup | Using istio smart DNS proxying | TCP traffic routed to BlackholeCluster

[Muthumj123]

meshConfig:
    defaultConfig:
      proxyMetadata:
        ISTIO_META_DNS_CAPTURE: "true"
        ISTIO_META_DNS_AUTO_ALLOCATE: "true"

Issue : After setting this up, Existing ServiceEntries (like aws dbs) for TCP traffic with resolution: DNS getting routed to BlackHoleCluster . If we change the resolution to NONE it works correctly.
Checked enpoints, listerners and even envoy config dump, could not track down why the behaviour is different.
Every issues or post that I read realted to this, says to set DNS VIP allocation for ServiceEntry but does not work for my case.
FYI. Outbout traffic is registry only

Is this a shorting coming for istio DNS? Any insight would help to understand this behaviour better.

Attaching access logs from proxy side car.

If resolution of serviceEntry is set to DNS, then access logs :

{
	"protocol": null,
	"upstream_local_address": null,
	"upstream_host": null,
	"authority": null,
	"start_time": "2024-01-18T21:42:47.347Z",
	"user_agent": null,
	"bytes_received": 0,
	"upstream_transport_failure_reason": null,
	"route_name": null,
	"request_id": null,
	"upstream_service_time": null,
	"response_code_details": null,
	"downstream_local_address": "172.xx.xx.xx:9142", #vpc private endpoint for db
	"x_forwarded_for": null,
	"method": null,
	"downstream_remote_address": "100.xx.xx.xx:36140", #podip
	"response_flags": "UH",
	"requested_server_name": null,
	"response_code": 0,
	"upstream_cluster": "BlackHoleCluster",
	"bytes_sent": 0,
	"duration": 11,
	"client_ip": null,
	"path": null
}

Istio pc listeners
ADDRESS=240.240.0.58, DESTINATION=Cluster: outbound|9142||cassandra.eu-west-1.amazonaws.com

If resolution of serviceEntry is set to NONE, then access logs looks like this and connects to the db.

{
	"method": null,
	"response_code": 0,
	"requested_server_name": null,
	"upstream_local_address": "100.1xx.xx.xx:60900", #podip
	"path": null,
	"protocol": null,
	"duration": 202,
	"upstream_host": "172.xx.xx.xx:9142", #vpc private endpoint for db
	"x_forwarded_for": null,
	"route_name": null,
	"bytes_received": 669,
	"upstream_cluster": "outbound|9142||cassandra.eu-west-1.amazonaws.com",
	"upstream_transport_failure_reason": null,
	"client_ip": null,
	"bytes_sent": 5698,
	"request_id": null,
	"user_agent": null,
	"response_code_details": null,
	"start_time": "2024-01-18T21:42:46.051Z",
	"upstream_service_time": null,
	"response_flags": "-",
	"downstream_remote_address": "100.xx.xx.xx:50350", #podip
	"downstream_local_address": "240.240.0.5:9142", # autoallocated VIP for the service entry
	"authority": null
}

[howardjohn]

I cannot fathom how NONE would work with auto_allocation. Basically that means the DNS response is going to return a bogus IP addrss (240.240.0.5), and envoy is going to pass the request through to that IP address. Since 240.240.0.5 is not as real IP its going to be broken.

DNS vs NONE mode should not impact traffic matching at all.

I feel there is maybe some information missing that explains this. Can you give more config like the Service entry, etc

See also https://istio.io/latest/docs/ops/configuration/traffic-management/traffic-routing/

[Muthumj123]

Our service entry looks like this. Many namspaces have same serviceentry but all of them are namespaced scope ( exportTo: - .)

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
spec:
  exportTo:
    - .
  hosts:
    - cassandra.eu-west-1.amazonaws.com
  location: MESH_EXTERNAL
  ports:
    - name: tcp
      number: 9142
      protocol: TCP
  resolution: DNS

And destination rule, like this :

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
spec:
  host: cassandra.eu-west-1.amazonaws.com
  trafficPolicy:
    tls:
      mode: DISABLE

[howardjohn]

Maybe you are running into #27619? I suspect the DNS proxy is not hitting at all.

Debugging steps:

• curl localhost:15020/debug/ndsz inside proxy. Do you see cassandra with a 240.x.x.x IP?
• Annotate sidecar.istio.io/agentLogLevel: dns:debug. Are queries getting seen?
• Get root into the pod network namespace (kubectl debug --image istio/base --profile sysadmin --attach -t -i if you have kubectl 1.30 prerelease!) and run watch -n1 iptables -t nat -v -n -L. Does the -A OUTPUT -d 10.96.0.10/32 -p udp -m udp --dport 53 -j REDIRECT --to-ports 15053 get hit?
• Add iptables log iptables -t nat -I OUTPUT 1 -j LOG --log-prefix "dns: ", check output with dmesg and echo 1 | sudo tee /proc/sys/net/netfilter/nf_log_all_netns. Does it show DST=kube-dns (service, not a specific pod -- if its a pod it is the cilium issue)? Does it show udp? does it show dport 53?

[Muthumj123]

@howardjohn Apologies for late response on this. We do not use cilinium, we use calico.
Please find my response inline.

curl localhost:15020/debug/ndsz inside proxy. Do you see cassandra with a 240.x.x.x IP?

		"cassandra.eu-west-1.amazonaws.com": {
			"ips": [
				"240.240.0.5"
			],
			"registry": "External"
		},
		"cassandra.eu-west-1.amazonaws.com.": {
			"ips": [
				"",
				"240.240.0.5"
			],
			"registry": "Kubernetes"
		},

Annotate sidecar.istio.io/agentLogLevel: dns:debug. Are queries getting seen?

{"level":"debug","time":"2024-02-06T14:48:22.326778Z","scope":"dns","msg":"request ;; opcode: QUERY, status: NOERROR, id: 9153\n;; flags: rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0\n\n;; QUESTION SECTION:\n;cassandra.eu-west-1.amazonaws.com.mynamespace.svc.cluster.local.\tIN\t A\n","protocol":"udp","edns":false,"id":"36528368-9c27-48e6-82b7-cae037e41c53"}

{"level":"debug","time":"2024-02-06T14:48:22.326852Z","scope":"dns","msg":"response for hostname \"cassandra.eu-west-1.amazonaws.com.mynamespace.svc.cluster.local.\" (found=true): ;; opcode: QUERY, status: NOERROR, id: 9153\n;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0\n\n;; QUESTION SECTION:\n;cassandra.eu-west-1.amazonaws.com.mynamespace.svc.cluster.local.\tIN\t A\n\n;; ANSWER SECTION:\ncassandra.eu-west-1.amazonaws.com.mynamespace.svc.cluster.local.\t30\tIN\tCNAME\tcassandra.eu-west-1.amazonaws.com.\ncassandra.eu-west-1.amazonaws.com.\t30\tIN\tA\t240.240.0.5\n","protocol":"udp","edns":false,"id":"36528368-9c27-48e6-82b7-cae037e41c53"}
{"level":"debug","time":"2024-02-06T14:48:22.326792Z","scope":"dns","msg":"request ;; opcode: QUERY, status: NOERROR, id: 26572\n;; flags: rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0\n\n;; QUESTION SECTION:\n;cassandra.eu-west-1.amazonaws.com.mynamespace.svc.cluster.local.\tIN\t AAAA\n","protocol":"udp","edns":false,"id":"f52a1160-c01e-4193-8ce3-22161536675b"}

{"level":"debug","time":"2024-02-06T14:48:22.326922Z","scope":"dns","msg":"response for hostname \"cassandra.eu-west-1.amazonaws.com.mynamespace.svc.cluster.local.\" (found=true): ;; opcode: QUERY, status: NOERROR, id: 26572\n;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0\n\n;; QUESTION SECTION:\n;cassandra.eu-west-1.amazonaws.com.mynamespace.svc.cluster.local.\tIN\t AAAA\n","protocol":"udp","edns":false,"id":"f52a1160-c01e-4193-8ce3-22161536675b"}

Get root into the pod network namespace (kubectl debug --image istio/base --profile sysadmin --attach -t -i if you have kubectl 1.30 prerelease!) and run watch -n1 iptables -t nat -v -n -L. Does the -A OUTPUT -d 10.96.0.10/32 -p udp -m udp --dport 53 -j REDIRECT --to-ports 15053 get hit?

• For this, Ephemeral pod did not work (always showing permission denied), hence I tried to get iptables output from the node where rhe pod is hosted. I could not see -A OUTPUT -d 10.96.0.10/32 -p udp -m udp --dport 53 -j REDIRECT --to-ports 15053 getting hit.

Add iptables log iptables -t nat -I OUTPUT 1 -j LOG --log-prefix "dns: ", check output with dmesg and echo 1 | sudo tee /proc/sys/net/netfilter/nf_log_all_netns. Does it show DST=kube-dns (service, not a specific pod -- if its a pod it is the cilium issue)? Does it show udp? does it show dport 53?

• dmesg shows pod ip as DST

[howardjohn]

dmesg shows pod ip as DST

[howardjohn]

dmesg shows pod ip as DST

This is the problem. Its not just a Cilium issue I guess, see #49284

[Muthumj123]

@howardjohn
Also noticed this discrepancy , do you think its related?

• We had implemeted multicluster using multi primary - multi network mesh.
• When doing istioctl pc endpoints IPs for remote cluster endpoints should be eastwest gateway IP, instead podIps from remote cluster are poplulated. hence not routable.
• Not consistent, but some proxies from same namespace showing correct ips.
• If I restart the pods, then the IPs are correct

[Muthumj123]

https://github.com/istio/istio/issues/49412

[rootsongjc]

This issue does not exist anymore.