Istio canary upgrade doesn't work as the doc

[hasakura12]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

I tried to perform canary upgrade on data plane using stable revision label: https://istio.io/latest/docs/setup/upgrade/canary/#stable-revision-labels

Here are outputs

# upgrade from v1.20 to 1.21
cd istio-${ISTIO_VERSION}/
export PATH=$PWD/bin:$PATH
istioctl version

$ istioctl install  \
    -f overrides.yaml  \
    --set values.global.tag=1.20.0 \
    --revision=1-20-0

$ istioctl tag set prod-stable --revision 1-20-0
Revision tag "prod-stable" created, referencing control plane revision "1-20-0". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable'

# also install v1.21.0 in parallel
ISTIO_VERSION=1.21.0
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=${ISTIO_VERSION} TARGET_ARCH=x86_64 sh -
cd istio-${ISTIO_VERSION}/
export PATH=$PWD/bin:$PATH
istioctl version

client version: 1.21.0
istiod version: 1.20.0
pilot version: 1.19.0
data plane version: 1.20.0 (6 proxies)

cd ../

$ istioctl install  \
    -f overrides.yaml  \
    --set values.global.tag=1.21.0 \
    --revision=1-21-0

$ istioctl tag set prod-canary --revision 1-21-0

$ istioctl tag list
TAG         REVISION NAMESPACES
prod-stable 1-20-0   
prod-canary 1-21-0   

kubectl get po,svc -n istio-system -l app=istio-egressgateway

# check two versions of istid deployed
$ kubectl get po,svc -n istio-system -l app=istiod
NAME                                 READY   STATUS    RESTARTS      AGE
pod/istiod-75ff4b6c4d-r8566          1/1     Running   3 (11d ago)   96d
pod/istiod-canary-7f95c9956b-4dzpg   1/1     Running   0             56s # <------ canary

NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                 AGE
service/istiod          ClusterIP   172.20.235.85   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP   303d
service/istiod-canary   ClusterIP   172.20.24.46    <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP   55s # <------ canary

$ kubectl get mutatingwebhookconfigurations
NAME                            WEBHOOKS   AGE
datadog-webhook                 3          303d
istio-revision-tag-default      4          303d
istio-sidecar-injector          4          303d
istio-sidecar-injector-canary   2          43s # <------ canary
pod-identity-webhook            1          348d
vpc-resource-mutating-webhook   1          348d


# Commands for Data Plane
istioctl proxy-status | grep "$(kubectl -n istio-system get pod -l app=istio-ingressgateway -o jsonpath='{.items..metadata.name}')" | awk '{print $10}'

$ k create ns stage-v1-21-0
namespace/stage-v1-21-0 created

$ kubectl label ns prod istio.io/rev=prod-stable
$ k describe ns prod

$ kubectl label ns stage-v1-21-0 istio.io/rev=prod-canary

kubectl label namespace stage-v1-21-0 istio-injection=enabled


$ k describe ns stage-v1-21-0
Name:         stage-v1-21-0
Labels:       istio.io/rev=prod-canary
              kubernetes.io/metadata.name=stage-v1-21-0

This somehow doesn't work, despite following the doc

$  istioctl ps
NAME                                                           CLUSTER        CDS        LDS        EDS        RDS          ECDS         ISTIOD                             VERSION
istio-egressgateway-5645b5c95d-hjfzr.istio-system              Kubernetes     SYNCED     SYNCED     SYNCED     NOT SENT     NOT SENT     istiod-1-21-0-5d6c57cdf6-ngtkf     1.21.0
istio-ingressgateway-848d667689-q5qdl.istio-system             Kubernetes     SYNCED     SYNCED     SYNCED     NOT SENT     NOT SENT     istiod-1-21-0-5d6c57cdf6-ngtkf     1.21.0
istio-private-ingressgateway-7644fb548d-8gkwq.istio-system     Kubernetes     SYNCED     SYNCED     SYNCED     SYNCED       NOT SENT     istiod-1-21-0-5d6c57cdf6-ngtkf     1.21.0

# doesn't work. Ref: https://istio.io/latest/docs/setup/upgrade/canary/#default-tag
$ istioctl tag set prod-stable --revision 1-21-1 --overwrite
Error: cannot modify tag: cannot find MutatingWebhookConfiguration with revision "1-21-1"

### Version

```Text
$ istioctl version
client version: 1.21.0
istiod version: 1.20.0
pilot version: 1.19.0
data plane version: 1.20.0 (6 proxies)

Additional Information

No response

[leosarra]

I see that you are calling the failed istioctl cmd with revision 1-21-1 instead of 1-21-0 that you have created earlier. Is that intentional?

[hasakura12]

I see that you are calling the failed istioctl cmd with revision 1-21-1 instead of 1-21-0 that you have created earlier. Is that intentional?

Wasn't intentional, but would it make a difference?

[leosarra]

Yes because a mutatingwebhook with revision 1-21-1 will not exist otherwise. The webhook is crated in the previous step that you executed with revision 1-21-0

[hasakura12]

Ok I see.

Btw, would you also look into & respond to my other open issue: #50597?