-
Notifications
You must be signed in to change notification settings - Fork 801
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
http experiments finish the pod service not recovery #4357
Comments
HI @howtosolveit , could you provide the log of chaos-daemon? it could help a lot for profiling. |
Sorry for providing the logs very late ; here is the log Chaos-daemon Version: version.Info{GitVersion:"chart-2.6.3", GitCommit:"b159d1c5a361ca350ca7789c9eb7fc917479db39", BuildDate:"2024-01-23T15:04:55Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"} �[2m2024-03-11T02:31:10.611521Z�[0m �[34mDEBUG�[0m �[1mhandle�[0m�[1m{�[0m�[3mproxy�[0m�[2m=�[0mProxy { opt: ProxyOpt { ipc_path: "/tmp/338cb870-f144-43c8-927b-fd056a885b7f.sock", verbose: 2 }, net_env: NetEnv { netns: "05f42e28-e91ens", device: "eth0", ip: "targetip/32", bridge1: "05f42e28-e91eb1", bridge2: "05f42e28-e91eb2", veth1: "05f42e28-e91ev1", veth2: "veth0", veth3: "veth1", veth4: "05f42e28-e91ev4", save_routes: [RouteMessage { header: RouteHeader { address_family: 2, destination_prefix_length: 32, source_prefix_length: 0, tos: 0, table: 254, protocol: 3, scope: 253, kind: 1, flags: (empty) }, nlas: [Table(254), Destination([169, 254, 1, 1]), Oif(3)] }, RouteMessage { header: RouteHeader { address_family: 2, destination_prefix_length: 0, source_prefix_length: 0, tos: 0, table: 254, protocol: 3, scope: 0, kind: 1, flags: (empty) }, nlas: [Table(254), Gateway([169, 254, 1, 1]), Oif(3)] }] }, rtnl_handle: Handle(ConnectionHandle { requests_tx: UnboundedSender(Some(UnboundedSenderInner { inner: UnboundedInner { state: 9223372036854775808, message_queue: Queue { head: 0x55b8c57ea220, tail: UnsafeCell { .. } }, num_senders: 1, recv_task: AtomicWaker } })) }), sender: Some(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: false, is_tx_task_set: false } }) }), rx: Some(Receiver { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: false, is_tx_task_set: false } }) }), task: None } �[3mrequest�[0m�[2m=�[0mRequest { method: PUT, uri: /, version: HTTP/1.1, headers: {"host": "", "user-agent": "Go-http-client/1.1", "content-length": "125"}, body: Body(Streaming) }�[1m}�[0m�[2m:�[0m �[2mchaos_tproxy::proxy::net::bridge�[0m�[2m:�[0m stdout : net.ipv4.ip_forward = 1 �[2m2024-03-11T02:31:10.613536Z�[0m �[34mDEBUG�[0m �[1mhandle�[0m�[1m{�[0m�[3mproxy�[0m�[2m=�[0mProxy { opt: ProxyOpt { ipc_path: "/tmp/338cb870-f144-43c8-927b-fd056a885b7f.sock", verbose: 2 }, net_env: NetEnv { netns: "05f42e28-e91ens", device: "eth0", ip: "targetip/32", bridge1: "05f42e28-e91eb1", bridge2: "05f42e28-e91eb2", veth1: "05f42e28-e91ev1", veth2: "veth0", veth3: "veth1", veth4: "05f42e28-e91ev4", save_routes: [RouteMessage { header: RouteHeader { address_family: 2, destination_prefix_length: 32, source_prefix_length: 0, tos: 0, table: 254, protocol: 3, scope: 253, kind: 1, flags: (empty) }, nlas: [Table(254), Destination([169, 254, 1, 1]), Oif(3)] }, RouteMessage { header: RouteHeader { address_family: 2, destination_prefix_length: 0, source_prefix_length: 0, tos: 0, table: 254, protocol: 3, scope: 0, kind: 1, flags: (empty) }, nlas: [Table(254), Gateway([169, 254, 1, 1]), Oif(3)] }] }, rtnl_handle: Handle(ConnectionHandle { requests_tx: UnboundedSender(Some(UnboundedSenderInner { inner: UnboundedInner { state: 9223372036854775808, message_queue: Queue { head: 0x55b8c57ea220, tail: UnsafeCell { .. } }, num_senders: 1, recv_task: AtomicWaker } })) }), sender: Some(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: false, is_tx_task_set: false } }) }), rx: Some(Receiver { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: false, is_tx_task_set: false } }) }), task: None } �[3mrequest�[0m�[2m=�[0mRequest { method: PUT, uri: /, version: HTTP/1.1, headers: {"host": "", "user-agent": "Go-http-client/1.1", "content-length": "125"}, body: Body(Streaming) }�[1m}�[0m�[2m:�[0m �[2mchaos_tproxy::proxy::net::bridge�[0m�[2m:�[0m stdout : net.ipv4.ip_nonlocal_bind = 1 �[2m2024-03-11T02:31:10.615575Z�[0m �[34mDEBUG�[0m �[1mhandle�[0m�[1m{�[0m�[3mproxy�[0m�[2m=�[0mProxy { opt: ProxyOpt { ipc_path: "/tmp/338cb870-f144-43c8-927b-fd056a885b7f.sock", verbose: 2 }, net_env: NetEnv { netns: "05f42e28-e91ens", device: "eth0", ip: "targetip/32", bridge1: "05f42e28-e91eb1", bridge2: "05f42e28-e91eb2", veth1: "05f42e28-e91ev1", veth2: "veth0", veth3: "veth1", veth4: "05f42e28-e91ev4", save_routes: [RouteMessage { header: RouteHeader { address_family: 2, destination_prefix_length: 32, source_prefix_length: 0, tos: 0, table: 254, protocol: 3, scope: 253, kind: 1, flags: (empty) }, nlas: [Table(254), Destination([169, 254, 1, 1]), Oif(3)] }, RouteMessage { header: RouteHeader { address_family: 2, destination_prefix_length: 0, source_prefix_length: 0, tos: 0, table: 254, protocol: 3, scope: 0, kind: 1, flags: (empty) }, nlas: [Table(254), Gateway([169, 254, 1, 1]), Oif(3)] }] }, rtnl_handle: Handle(ConnectionHandle { requests_tx: UnboundedSender(Some(UnboundedSenderInner { inner: UnboundedInner { state: 9223372036854775808, message_queue: Queue { head: 0x55b8c57ea220, tail: UnsafeCell { .. } }, num_senders: 1, recv_task: AtomicWaker } })) }), sender: Some(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: false, is_tx_task_set: false } }) }), rx: Some(Receiver { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: false, is_tx_task_set: false } }) }), task: None } �[3mrequest�[0m�[2m=�[0mRequest { method: PUT, uri: /, version: HTTP/1.1, headers: {"host": "", "user-agent": "Go-http-client/1.1", "content-length": "125"}, body: Body(Streaming) }�[1m}�[0m�[2m:�[0m �[2mchaos_tproxy::proxy::net::bridge�[0m�[2m:�[0m stdout : net.ipv4.conf.367ff703-7ce0b2.rp_filter = 0 �[2m2024-03-11T02:31:10.617931Z�[0m �[34mDEBUG�[0m �[1mhandle�[0m�[1m{�[0m�[3mproxy�[0m�[2m=�[0mProxy { opt: ProxyOpt { ipc_path: "/tmp/338cb870-f144-43c8-927b-fd056a885b7f.sock", verbose: 2 }, net_env: NetEnv { netns: "05f42e28-e91ens", device: "eth0", ip: "targetip/32", bridge1: "05f42e28-e91eb1", bridge2: "05f42e28-e91eb2", veth1: "05f42e28-e91ev1", veth2: "veth0", veth3: "veth1", veth4: "05f42e28-e91ev4", save_routes: [RouteMessage { header: RouteHeader { address_family: 2, destination_prefix_length: 32, source_prefix_length: 0, tos: 0, table: 254, protocol: 3, scope: 253, kind: 1, flags: (empty) }, nlas: [Table(254), Destination([169, 254, 1, 1]), Oif(3)] }, RouteMessage { header: RouteHeader { address_family: 2, destination_prefix_length: 0, source_prefix_length: 0, tos: 0, table: 254, protocol: 3, scope: 0, kind: 1, flags: (empty) }, nlas: [Table(254), Gateway([169, 254, 1, 1]), Oif(3)] }] }, rtnl_handle: Handle(ConnectionHandle { requests_tx: UnboundedSender(Some(UnboundedSenderInner { inner: UnboundedInner { state: 9223372036854775808, message_queue: Queue { head: 0x55b8c57ea220, tail: UnsafeCell { .. } }, num_senders: 1, recv_task: AtomicWaker } })) }), sender: Some(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: false, is_tx_task_set: false } }) }), rx: Some(Receiver { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: false, is_tx_task_set: false } }) }), task: None } �[3mrequest�[0m�[2m=�[0mRequest { method: PUT, uri: /, version: HTTP/1.1, headers: {"host": "", "user-agent": "Go-http-client/1.1", "content-length": "125"}, body: Body(Streaming) }�[1m}�[0m�[2m:�[0m �[2mchaos_tproxy::proxy::net::bridge�[0m�[2m:�[0m stdout : net.ipv4.conf.veth0.rp_filter = 0 �[2m2024-03-11T02:31:10.620383Z�[0m �[34mDEBUG�[0m �[1mhandle�[0m�[1m{�[0m�[3mproxy�[0m�[2m=�[0mProxy { opt: ProxyOpt { ipc_path: "/tmp/338cb870-f144-43c8-927b-fd056a885b7f.sock", verbose: 2 }, net_env: NetEnv { netns: "05f42e28-e91ens", device: "eth0", ip: "targetip/32", bridge1: "05f42e28-e91eb1", bridge2: "05f42e28-e91eb2", veth1: "05f42e28-e91ev1", veth2: "veth0", veth3: "veth1", veth4: "05f42e28-e91ev4", save_routes: [RouteMessage { header: RouteHeader { address_family: 2, destination_prefix_length: 32, source_prefix_length: 0, tos: 0, table: 254, protocol: 3, scope: 253, kind: 1, flags: (empty) }, nlas: [Table(254), Destination([169, 254, 1, 1]), Oif(3)] }, RouteMessage { header: RouteHeader { address_family: 2, destination_prefix_length: 0, source_prefix_length: 0, tos: 0, table: 254, protocol: 3, scope: 0, kind: 1, flags: (empty) }, nlas: [Table(254), Gateway([169, 254, 1, 1]), Oif(3)] }] }, rtnl_handle: Handle(ConnectionHandle { requests_tx: UnboundedSender(Some(UnboundedSenderInner { inner: UnboundedInner { state: 9223372036854775808, message_queue: Queue { head: 0x55b8c57ea220, tail: UnsafeCell { .. } }, num_senders: 1, recv_task: AtomicWaker } })) }), sender: Some(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: false, is_tx_task_set: false } }) }), rx: Some(Receiver { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: false, is_tx_task_set: false } }) }), task: None } �[3mrequest�[0m�[2m=�[0mRequest { method: PUT, uri: /, version: HTTP/1.1, headers: {"host": "", "user-agent": "Go-http-client/1.1", "content-length": "125"}, body: Body(Streaming) }�[1m}�[0m�[2m:�[0m �[2mchaos_tproxy::proxy::net::bridge�[0m�[2m:�[0m stdout : net.ipv4.conf.veth1.rp_filter = 0 �[2m2024-03-11T02:31:10.622649Z�[0m �[34mDEBUG�[0m �[1mhandle�[0m�[1m{�[0m�[3mproxy�[0m�[2m=�[0mProxy { opt: ProxyOpt { ipc_path: "/tmp/338cb870-f144-43c8-927b-fd056a885b7f.sock", verbose: 2 }, net_env: NetEnv { netns: "05f42e28-e91ens", device: "eth0", ip: "targetip/32", bridge1: "05f42e28-e91eb1", bridge2: "05f42e28-e91eb2", veth1: "05f42e28-e91ev1", veth2: "veth0", veth3: "veth1", veth4: "05f42e28-e91ev4", save_routes: [RouteMessage { header: RouteHeader { address_family: 2, destination_prefix_length: 32, source_prefix_length: 0, tos: 0, table: 254, protocol: 3, scope: 253, kind: 1, flags: (empty) }, nlas: [Table(254), Destination([169, 254, 1, 1]), Oif(3)] }, RouteMessage { header: RouteHeader { address_family: 2, destination_prefix_length: 0, source_prefix_length: 0, tos: 0, table: 254, protocol: 3, scope: 0, kind: 1, flags: (empty) }, nlas: [Table(254), Gateway([169, 254, 1, 1]), Oif(3)] }] }, rtnl_handle: Handle(ConnectionHandle { requests_tx: UnboundedSender(Some(UnboundedSenderInner { inner: UnboundedInner { state: 9223372036854775808, message_queue: Queue { head: 0x55b8c57ea220, tail: UnsafeCell { .. } }, num_senders: 1, recv_task: AtomicWaker } })) }), sender: Some(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: false, is_tx_task_set: false } }) }), rx: Some(Receiver { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: false, is_tx_task_set: false } }) }), task: None } �[3mrequest�[0m�[2m=�[0mRequest { method: PUT, uri: /, version: HTTP/1.1, headers: {"host": "", "user-agent": "Go-http-client/1.1", "content-length": "125"}, body: Body(Streaming) }�[1m}�[0m�[2m:�[0m �[2mchaos_tproxy::proxy::net::bridge�[0m�[2m:�[0m stdout : net.ipv4.conf.lo.rp_filter = 0 �[2m2024-03-11T02:31:10.624745Z�[0m �[34mDEBUG�[0m �[1mhandle�[0m�[1m{�[0m�[3mproxy�[0m�[2m=�[0mProxy { opt: ProxyOpt { ipc_path: "/tmp/338cb870-f144-43c8-927b-fd056a885b7f.sock", verbose: 2 }, net_env: NetEnv { netns: "05f42e28-e91ens", device: "eth0", ip: "targetip/32", bridge1: "05f42e28-e91eb1", bridge2: "05f42e28-e91eb2", veth1: "05f42e28-e91ev1", veth2: "veth0", veth3: "veth1", veth4: "05f42e28-e91ev4", save_routes: [RouteMessage { header: RouteHeader { address_family: 2, destination_prefix_length: 32, source_prefix_length: 0, tos: 0, table: 254, protocol: 3, scope: 253, kind: 1, flags: (empty) }, nlas: [Table(254), Destination([169, 254, 1, 1]), Oif(3)] }, RouteMessage { header: RouteHeader { address_family: 2, destination_prefix_length: 0, source_prefix_length: 0, tos: 0, table: 254, protocol: 3, scope: 0, kind: 1, flags: (empty) }, nlas: [Table(254), Gateway([169, 254, 1, 1]), Oif(3)] }] }, rtnl_handle: Handle(ConnectionHandle { requests_tx: UnboundedSender(Some(UnboundedSenderInner { inner: UnboundedInner { state: 9223372036854775808, message_queue: Queue { head: 0x55b8c57ea220, tail: UnsafeCell { .. } }, num_senders: 1, recv_task: AtomicWaker } })) }), sender: Some(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: false, is_tx_task_set: false } }) }), rx: Some(Receiver { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: false, is_tx_task_set: false } }) }), task: None } �[3mrequest�[0m�[2m=�[0mRequest { method: PUT, uri: /, version: HTTP/1.1, headers: {"host": "", "user-agent": "Go-http-client/1.1", "content-length": "125"}, body: Body(Streaming) }�[1m}�[0m�[2m:�[0m �[2mchaos_tproxy::proxy::net::bridge�[0m�[2m:�[0m stdout : net.ipv4.conf.all.rp_filter = 0 �[2m2024-03-11T02:31:10.632004Z�[0m �[32m INFO�[0m �[2mchaos_tproxy::proxy::uds_server�[0m�[2m:�[0m Uds listener listening on "/tmp/e53b05cf-3331-48b0-90c1-00b88520cf38.sock" |
Bug Report
What version of Kubernetes are you using?
aws eks v.1.29.0
What version of Chaos Mesh are you using?
Controller manager Version: version.Info{GitVersion:"chart-2.6.3", GitCommit:"b159d1c5a361ca350ca7789c9eb7fc917479db39", BuildDate:"2024-01-23T15:04:53Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
What did you do? / Minimal Reproducible Example
1 build deployment in default namespace
apiVersion: apps/v1 kind: Deployment metadata: name: httpd namespace: default labels: app: httpd spec: replicas: 1 selector: matchLabels: app: httpd template: metadata: labels: app: httpd spec: containers: - name: httpd image: centos/httpd ports: - containerPort: 8080 securityContext: privileged: true
2 config experiments
kind: HTTPChaos apiVersion: chaos-mesh.org/v1alpha1 metadata: namespace: default name: chaos-test-http-02 spec: selector: namespaces: - default labelSelectors: app: httpd mode: all target: Request abort: true port: 80 path: '*' method: POST duration: 5s
when the experiments finished, i found the request still not work
and i found the pod still has the tproxy process and inside the pod the request is ok
What did you expect to see?
the request recovery
The text was updated successfully, but these errors were encountered: