canary migration from operator to helm install

[cespeche]

Hello,
I'm trying to switch from the operator to helm without uninstalling the operator since it will delete the gateways. I've done a few canay upgrades using the operator and I only have versioned istiod.
I'm installing new istio and gateways with helm, once the new version is up and running I'll switch the DNS and delete the old version with the operator. Hopefully that is something achievable!

Istio operator version: 1.15.7
GKE version: 1.25.16-gke.1596000
Istio helm version: 1.20.4

% kubectl get iop -A
NAMESPACE      NAME                          REVISION   STATUS    AGE
istio-operator   istiooperator-1-15-7        1-15-7     HEALTHY   193d

% helm ls -n istio-system
NAME                	NAMESPACE   	REVISION	UPDATED                             	STATUS  	CHART        	APP VERSION
istio-base-default  	istio-system	3       	2024-04-10 10:06:16.944996 +0100 BST	deployed	base-1.20.4  	1.20.4     
istio-istiod-default	istio-system	8       	2024-04-10 10:07:16.839706 +0100 BST	deployed	istiod-1.20.4	1.20.4

% kubectl get mutatingwebhookconfiguration                                                                                                 
NAME                                                 WEBHOOKS   AGE
cert-manager-webhook                                 1          162d
gke-vpa-webhook-config                               1          15d
istio-revision-tag-default                           4          20h
istio-sidecar-injector-1-15-7                        2          193d
istio-sidecar-injector-1-20-4                        2          36m

Istio helm didn't create a new validatingwebhookconfiguration, it overwrites the istiod-default-validator

% kubectl get validatingwebhookconfigurations    
NAME                                                              WEBHOOKS   AGE
cert-manager-webhook                                              1          162d
flowcontrol-guardrails.config.common-webhooks.networking.gke.io   1          2y159d
gkepolicy.config.common-webhooks.networking.gke.io                1          318d
istio-validator-1-15-7-istio-system                               1          193d
istiod-default-validator                                          1          44h
nodelimit.config.common-webhooks.networking.gke.io                1          2y277d
validation-webhook.snapshot.storage.k8s.io                        1          2y159d

% kubectl get validatingwebhookconfigurations istiod-default-validator -o yaml -o jsonpath='{.webhooks[0].clientConfig.service}'
{"name":"istiod-1-20-4","namespace":"istio-system","path":"/validate","port":443

% istioctl tag list                    
TAG     REVISION NAMESPACES
default 1-20-4   test,istio-system

% kubectl run -n test busybox --image=busybox -- sh -c "while true; do sleep infinity; done"
Error from server (InternalError): Internal error occurred: failed calling webhook "rev.namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod-1-20-4.istio-system.svc:443/inject?timeout=10s": tls: failed to verify certificate: x509: certificate is valid for istiod.istio-system.svc, not istiod-1-20-4.istio-system.svc

It works if I point the default tag to 1-15-7

% kubectl logs -n istio-system deploy/istiod-1-20-4
2024-04-10T09:13:25.934315Z	info	ads	Push debounce stable[11] 1 for config ServiceEntry/istio-system/istiod-1-15-7.istio-system.svc.cluster.local: 100.340537ms since last change, 100.340359ms since last push, full=false
2024-04-10T09:13:25.934395Z	info	ads	XDS: Incremental Pushing ConnectedEndpoints:0 Version:2024-04-10T09:07:02Z/2
2024-04-10T09:13:26.183607Z	info	ads	Push debounce stable[12] 1 for config ServiceEntry/istio-system/istiod-1-15-7.istio-system.svc.cluster.local: 100.350484ms since last change, 100.350309ms since last push, full=false
2024-04-10T09:13:26.183688Z	info	ads	XDS: Incremental Pushing ConnectedEndpoints:0 Version:2024-04-10T09:07:02Z/2
2024-04-10T09:13:44.284930Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2024-04-10T09:13:44.284964Z	info	validationController	validatingwebhookconfiguration istiod-default-validator (failurePolicy=Ignore, resourceVersion=1397824568) is up-to-date. No change required.
2024-04-10T09:13:44.284975Z	error	controllers	error handling istiod-default-validator, retrying (retry count: 17): webhook is not ready, retry	controller=validation
2024-04-10T09:14:44.294375Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2024-04-10T09:14:44.294416Z	info	validationController	validatingwebhookconfiguration istiod-default-validator (failurePolicy=Ignore, resourceVersion=1397824568) is up-to-date. No change required.
2024-04-10T09:14:44.294434Z	error	controllers	error handling istiod-default-validator, retrying (retry count: 18): webhook is not ready, retry	controller=validation
... <repeated output truncated>...
2024-04-10T09:46:44.662639Z	error	controllers	error handling istiod-default-validator, retrying (retry count: 50): webhook is not ready, retry	controller=validation
2024-04-10T09:47:31.214848Z	info	http: TLS handshake error from 172.19.0.8:46694: remote error: tls: bad certificate
2024-04-10T09:47:44.673794Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2024-04-10T09:47:44.673830Z	info	validationController	validatingwebhookconfiguration istiod-default-validator (failurePolicy=Ignore, resourceVersion=1397824568) is up-to-date. No change required.
2024-04-10T09:47:44.673840Z	error	controllers	error handling istiod-default-validator, retrying (retry count: 51): webhook is not ready, retry	controller=validation
2024-04-10T09:48:44.685912Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2024-04-10T09:48:44.685941Z	info	validationController	validatingwebhookconfiguration istiod-default-validator (failurePolicy=Ignore, resourceVersion=1397824568) is up-to-date. No change required.
2024-04-10T09:48:44.685954Z	error	controllers	error handling istiod-default-validator, retrying (retry count: 52): webhook is not ready, retry	controller=validation

The caBundle doesn't match what is mounted in the mutatingwebhooksconfiguration but it works for 1-15-7

% kubectl -n istio-system get configmap istio-ca-root-cert
NAME                 DATA   AGE
istio-ca-root-cert   1      2y299d
% kubectl -n istio-system get configmap istio-ca-root-cert -o jsonpath='{.data.root-cert\.pem}' | base64 | md5sum
e871385336bd78d1c59980c3661622b2  -
% kubectl get mutatingwebhookconfiguration istio-sidecar-injector-1-20-4 -o yaml -o jsonpath='{.webhooks[0].clientConfig.caBundle}'| md5sum
066d750196cc4ffca4b3300d66bc6628  -
% kubectl get mutatingwebhookconfiguration istio-sidecar-injector-1-15-7 -o yaml -o jsonpath='{.webhooks[0].clientConfig.caBundle}'| md5sum
066d750196cc4ffca4b3300d66bc6628  -

I've tried installing version 1.17.8 and I got the same error.
Any help is highly appreciated, thank you!