[Question] Unable to do "kubectl logs" for pods running in edge node. #1838
Comments
|
@chunfungintel Hi, I think you should deploy Raven like this to enable node IP forward: After that, you need to create the Gateway CR, see here |
|
Hi, Thank you for your suggestion. I modified my steps as below:
Unfortunately, I still unable to do 'kubectl logs' on edge node successfully. Any idea yet? :) |
|
@chunfungintel I think you should use v0.3.2 instead of v0.4 for raven's image version if you are still deploying v1.3 openyurt |
|
Hi @YTGhost Actually, these are the only available versions available in helm I do not need to specifically need to use v1.3 OpenYurt, do you have any version that I should try on? It seems in specific version, Raven controller is merged into yurt-manager(correct me if I am wrong), is that a version before 1.3? |
@chunfungintel raven's previous version of Chart doesn't look like managed very well, I think you can use openyurt v1.4 since v0.4 raven upgraded the CRD. Of course you can also use openyurt v1.3, maybe you have to manually change raven's Chart package. For example, using version 0.1.1 of Chart and manually adjusting the image version of raven-agent to v0.3.2.
We merged raven-controller-manager into yurt-manager in v1.3, so in v1.3 and beyond, you only need to install yurt-manager. |
|
Revised steps: Control-panel initialization: sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12
mkdir -p $HOME/.kube && sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config && sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
kubectl taint nodes --all node-role.kubernetes.io/master-Using OpenYurt 1.4.0 + Raven agent 0.4.0 helm upgrade --install yurt-manager -n kube-system openyurt/yurt-manager --version 1.4.0 --set image.tag=latest
helm upgrade --install yurt-hub -n kube-system --set kubernetesServerAddr=https://${KUBERNETES_SERVER_ADDRESS}:6443 openyurt/yurthub --version 1.4.0
helm upgrade --install raven-agent -n kube-system openyurt/raven-agent --set vpn.forwardNodeIP=true \
--set image.tag=0.4.0 --version 0.4.0Install OpenYurt 1.4 in Edge wget https://github.com/openyurtio/openyurt/releases/download/v1.4.0/yurtadm-v1.4.0-linux-amd64.tar.gz
tar -xvf yurtadm-v1.4.0-linux-amd64.tar.gz
sudo cp linux-amd64/yurtadm /usr/local/bin/yurtadm && sudo chmod +x /usr/local/bin/yurtadmEdge node joining: sudo yurtadm join \
${CONTROL_PANEL_ADDRESS}:6443 \
--token=${JOIN_TOKEN} --node-type=edge \
--cri-socket=unix:///run/containerd/containerd.sock \
--discovery-token-ca-cert-hash=${CA_HASH} --v=5Gateway configuration: kubectl label nodes adl-edge-node raven.openyurt.io/gateway=gw-edge; \
kubectl label nodes adl-cloud-node raven.openyurt.io/gateway=gw-cloud
cat <<EOF | kubectl apply -f -
apiVersion: raven.openyurt.io/v1alpha1
kind: Gateway
metadata:
name: gw-edge
spec:
endpoints:
- nodeName: adl-edge-node
underNAT: true
---
apiVersion: raven.openyurt.io/v1alpha1
kind: Gateway
metadata:
name: gw-cloud
spec:
endpoints:
- nodeName: adl-cloud-node
underNAT: false
EOF
git clone https://github.com/openyurtio/raven.git
cd raven && git checkout v0.4.0
make deployResults: Anything still missing? |
@chunfungintel Hi, could you please provide the logs of raven-agent? |
|
@YTGhost W1208 03:07:36.262826 1 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I1208 03:07:37.395489 1 libreswan.go:385] start pluto successfully It seems to me the configuration failed due to I am behind cooperate proxy? |
@chunfungintel I think it should be, raven will go to the public network and request to get the PublicIp, however maybe it's because of your network environment, there was a problem with the request process. If there is no way to get it automatically, you can also get it manually and set the |
|
@chunfungintel Hi, has this been resolved or any progress made? |
|
@YTGhost Actually I was collecting logs when you asking :) What I do currently is inject http_proxy, https_proxy and no_proxy with Raven's logs from control panel: Raven's logs from edge node(grabbed from /var/log/pods/kube-system_raven-agent-ds) what i am thinking with last line of Edge's Raven's last line of log is it is routed to a proxy, what no_proxy not captured. I had other observation too, sharing in other post. |
|
Another observation I noticed after setting up gateway, is the nodes became "nonready" shortly after From YurtHub logs, it failed to connect to the control-panel: |
|
@YTGhost Can I know how to do this? |
@chunfungintel |
About How to get PublicIp manually, you can use some public API to get it, for example, https://ifconfig.me/. |
Please refer to the document https://openyurt.io/zh/docs/next/user-manuals/network/raven , you can set the field spec.endpoints.publicIP = 129.xxx.xxx.xxx |
|
This is my testing topology and gateway configuration, please advice. graph
B("Control-Panel (adl-cloud-node)")
B ---|10.226.xx.xx/23| C{Router}
C ---|192.168.1.100/24| D["Edge (adl-edge-node)"]
C ---|192.168.1.200/24| E["Edge (adl-edge-node-2)"]
Logs from raven in edge node: Obviously, my cooperate network blocking used of stun, checking with pystun3: |
|
Update: apiVersion: raven.openyurt.io/v1beta1
kind: Gateway
metadata:
name: gw-cloud
spec:
exposeType: PublicIP
endpoints:
- nodeName: adl-cloud-node
port: 4500
type: tunnel
publicIP: LOCAL_NETWORK_IP
proxyConfig:
Replicas: 1
tunnelConfig:
Replicas: 1
EOF
---
apiVersion: raven.openyurt.io/v1alpha1
kind: Gateway
metadata:
name: gw-edge
spec:
endpoints:
- nodeName: adl-edge-node
underNAT: trueAND set correct proxy settings in raven-agent-ds Thanks a lot for yours support! |
You don't need this complicated configuration, you just need to enable Raven's Tunnel mode and configure the correct Gateway CR https://openyurt.io/zh/docs/user-manuals/network/raven/ and yurt-manager will elect activeEndpoints in Gateway.Status.ActiveEndpoints. You can |
|
@River-sh Thank you, I will try and let you know. |
|
Troubled by same question several days. Maybe a bit different network environments from @chunfungintel . How could I configure Gateway CR correctly when both control-plane nodes and edge nodes are behind NAT? |
You can choose to expose the gateway node of the control plane on the public network (configure DNAT on the NAT so that the UDP 4500 of this gateway node can be accessed), and the Gateway is set to UnderNAT=false. You can also set underNat = true to test whether NAT traversal is implemented to build a VPN between two gateway nodes. You can let raven-agent enable nat traversal，but not all NATs can be traversed |
|
I used the same revised step except that raven-agent-0.4.1 was used.
|
As you said, your cloud nodes cannot be accessed on the public network, cross-network domain VPNs cannot be established, and can not use kubectl logs/exec |
|
@qpanpony You can read this document step by step. https://openyurt.io/zh/docs/user-manuals/network/raven |
|
Just a feedback. I quitted to use raven-agent component since cross-network domain VPNs cannot be established under my network environment. Have deployed edgemesh which provided the ability to communicate across subnets based on LibP2P tunnel. |
What happened:
Unable to do "kubectl logs" for pods in edge node.
What you expected to happen:
Success to view logs in edge node.
How to reproduce it (as minimally and precisely as possible):
Control-panel setup:
Kubernetes version:
kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-02-22T13:33:14Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
Kubernete initialization:
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
OpenYurt installation:
helm upgrade --install yurt-manager -n kube-system openyurt/yurt-manager --version 1.3.4
helm upgrade --install yurt-hub -n kube-system --set kubernetesServerAddr=https://${KUBERNETES_SERVER_ADDRESS}:6443 openyurt/yurthub --version 1.3.4
helm upgrade --install raven-agent -n kube-system openyurt/raven-agent
Edge node:
Installation:
sudo rm
which kubelet kubeadm kubectlwget https://github.com/openyurtio/openyurt/releases/download/v1.3.4/yurtadm-v1.3.4-linux-amd64.zip
unzip yurtadm-v1.3.4-linux-amd64.zip
sudo cp linux-amd64/yurtadm /usr/local/bin/yurtadm &&
sudo chmod +x /usr/local/bin/yurtadm
Joining:
sudo yurtadm join
${CONTROL_PANEL_ADDRESS}:6443
--token=${JOIN_TOKEN} --node-type=edge
--cri-socket=unix:///run/containerd/containerd.sock
--discovery-token-ca-cert-hash=${CA_HASH} --v=5
Anything else we need to know?:
Control panel node in subnet 10.226.76.0/23, while edge node in 192.168.0.0/24.
I am able to join and deploy workload, but failed to view its logs.
I am not sure which steps I missed?
Environment:
kubectl version): 1.23.17cat /etc/os-release): Ubuntu 22.04.3 LTSuname -a): 6.2.0-37-genericothers
/kind question
The text was updated successfully, but these errors were encountered: