shadow api can not access pv which dynamicly created by pvc

[x-7]

What happened:

when use clusternet create a pvc without specfic volume,the pv dynamically created by the pvc can not be access with shandow api.

What you expected to happen:

expect the resouce created by it's parent resource(which is create with clusternet ) can be access with shandow api

How to reproduce it (as minimally and precisely as possible):

pvcbug.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: foo
---
apiVersion: v1
kind: Pod
metadata:
 name: pod1
 namespace: foo
spec:
 containers:
  - image: nginx:1.14.2
    name: pod1
    volumeMounts:
      - name: azure
        mountPath: /mnt/azure
 volumes:
  - name: azure
    persistentVolumeClaim:
      claimName: pvc1
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc1
  namespace: foo
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  volumeMode: Filesystem
  storageClassName: azurefile-csi
---
apiVersion: apps.clusternet.io/v1alpha1
kind: Subscription
metadata:
  name: subs1
  namespace: default
spec:
  subscribers:
    - clusterAffinity:
        matchLabels:
          clusters.clusternet.io/cluster-id: {your cluster id}
  feeds:
    - apiVersion: v1
      kind: Namespace
      name: foo
    - apiVersion: v1
      kind: Pod
      name: pod1
      namespace: foo
    - apiVersion: v1
      kind: PersistentVolumeClaim
      name: pvc1

0. prepare azure aks environment (https://learn.microsoft.com/en-us/azure/aks/azure-csi-files-storage-provision)
1. kubectl clusternet apply -f ./pvcbug.yaml
2. find the pv dynamic created
3. kubectl clusternet get pv |grep {the pv's name in step2}
   expect: the pv will appear in the output
   actual : the output is empty

Anything else we need to know?:

Environment:

• Clusternet version: latest
• k8s Cluster : azure aks

[dixudx]

@x-7 Would you please share the steps? So we could reproduce what happened. Thanks.

[dixudx]

1. kubectl clusternet apply -f ./pvcbug.yaml
2. find the pv dynamic created
3. kubectl clusternet get pv |grep {the pv's name in step2}
   expect: the pv will appear in the output
   actual : the output is empty

@x-7 What kubectl clusternet does is creating a template, not a real resource. Thus kubectl clusternet get and kubectl get have different outputs.

Also pv will not be auto-created in kubectl clusternet view. Learn more from Resources Creation in Clusternet.

[x-7]

1. kubectl clusternet apply -f ./pvcbug.yaml
2. find the pv dynamic created
3. kubectl clusternet get pv |grep {the pv's name in step2}
   expect: the pv will appear in the output
   actual : the output is empty

@x-7 What kubectl clusternet does is creating a template, not a real resource. Thus kubectl clusternet get and kubectl get have different outputs.

Also pv will not be auto-created in kubectl clusternet view. Learn more from Resources Creation in Clusternet.
i think that all necessary resource info should be avaliable with shadowApi when i created resource with subscriptions。
but only the template(manifests) and status( in Descriptions) avaliable with shadowApi。
other info is also necessary,for example :

1. spec.volumeName field in pvc.
2. the pv auto created by pvc
3. Annotation information after resource creation.

I think one of the original reasons for designing shadowapi was to seamlessly replace the k8s native api in the business logic, so it only makes sense that at least what the k8s native api can access, the k8s shandowApi should also be able to access。

If not, I can only get the appropriate cluster id from the subscription, get the appropriate k8s configuration based on the cluster id, and then use the native api to get the resource information. This bypass looks very unclusternet

[dixudx]

@x-7 Clusternet is used for multi-cluster scenario. It helps deploy these template resources (the so-called shadow api resources) to child clusters.

For these shadow resources that are not deployed to child clusters, it is not desired to populate any kinds of other resources in parent cluster. Taking pvc as an example. We want this PVC to be deployed in child clusters, right? Then what's the use of having a PV for this PVC in parent cluster? That does not make sense. When this PVC is deployed to a child cluster, the pv provider/provisioner in the child cluster will prepare and bind a proper PV for such a PVC.

Moreover, in different child clusters, even a same named PVC can be bound with different named PVs. We can not make sure all these PVs are identical. That is totally impossible. These PVs may be from any kinds of storage providers as well.

I think Resources Creation in Clusternet can show you some backgrounds and the design purposes.

[x-7]

1. "Then what's the use of having a PV for this PVC in parent cluster? That does not make sense"
   yes, the pv in parent cluster is no use in clusternet。but what i need is to get the pv info by the pvc, so i can upload some file to the pv in my business scenario, and i just want to do the thing with shadowApi rather than with original k8sApi. becasue my app is also multi-cluster scenario. I don't want to switch back and forth between k8s connection configurations for multiple clusters。

2. "Moreover, in different child clusters, even a same named PVC can be bound with different named PVs. We can not make sure all these PVs are identical. That is totally impossible. These PVs may be from any kinds of storage providers as well.
   " A pvc can only be bound to one pv at a time. This pvc can also be bound to a different pv。i can not understand why you care "these PVs are identical". when the bouded pv changed , if we can get the new bind pv ,that's ok.

3. "I think Resources Creation in Clusternet can show you some backgrounds and the design purposes"
   great design, but from a usage point of view, I am concerned about being able to retrieve any information I need solely through the Shadow API. I prefer not to switch between multiple Kubernetes clusters for connectivity.
   I think this is a common need for the majority of users when using the Shadow API.

[dixudx]

but what i need is to get the pv info by the pvc, so i can upload some file to the pv in my business scenario, and i just want to do the thing with shadowApi rather than with original k8sApi. becasue my app is also multi-cluster scenario. I don't want to switch back and forth between k8s connection configurations for multiple clusters

@x-7 I understand your user scenarios. It's painful to switch k8s contexts back and forth.

We had such scenario as well. But there are multiple aspects that we need to consider before we move next.

1. How could we specify or inline cluster context? Such as kubectl clusternet --cluster-id=xxx ? What the request url will be like? And how to align with native k8s style?
2. Should we incorporate these status/metadata/info/labels/annotations in a single manifest object or separate objects?
3. Should we pass every single change from child clusters to the parent cluster?
4. Some potential security issues. We don't want cluster-specific data be leaked to other unrelated users. This also affects the data model (question 2) for this scenario.
5. HUGE DATA. And they are duplicated. This may overwhelm the parent kube-apiserver. And lots of frequent updates. If we've got 10 clusters, 100 clusters, or more, the data dimension is horrendous.
6. ...

[dixudx]

Currently we've made some attempts to aggregate resource status from child clusters. Only jobs/statefulsets/deployments workloads are supported right now.

[x-7]

"How could we specify or inline cluster context? Such as kubectl clusternet --cluster-id=xxx ? What the request url will be like? And how to align with native k8s style?"
How do you view this solution: https://open-cluster-management.io/scenarios/pushing-kube-api-requests/ ?
Is there a way for clusternet to use this solution?
shadowApi + ProxyApi
The combination of these two can cover all usage needs, although not very elegantly.

[x-7]

but what i need is to get the pv info by the pvc, so i can upload some file to the pv in my business scenario, and i just want to do the thing with shadowApi rather than with original k8sApi. becasue my app is also multi-cluster scenario. I don't want to switch back and forth between k8s connection configurations for multiple clusters

@x-7 I understand your user scenarios. It's painful to switch k8s contexts back and forth.

We had such scenario as well. But there are multiple aspects that we need to consider before we move next.

1. How could we specify or inline cluster context? Such as kubectl clusternet --cluster-id=xxx ? What the request url will be like? And how to align with native k8s style?
2. Should we incorporate these status/metadata/info/labels/annotations in a single manifest object or separate objects?
3. Should we pass every single change from child clusters to the parent cluster?
4. Some potential security issues. We don't want cluster-specific data be leaked to other unrelated users. This also affects the data model (question 2) for this scenario.
5. HUGE DATA. And they are duplicated. This may overwhelm the parent kube-apiserver. And lots of frequent updates. If we've got 10 clusters, 100 clusters, or more, the data dimension is horrendous.
6. ...

It feels so difficult