v1.9.0-rc chaos-mesh case failed

[cyliu0]

Describe the bug

https://buildkite.com/risingwave-test/chaos-mesh/builds/826#018f659e-814b-43fd-9534-32cadf65a1a4
https://buildkite.com/risingwave-test/chaos-mesh/builds/825#018f659e-81e8-42f4-8a0d-238aa91fc23b

Seems like the test stuck at creating table

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

v1.9.0-rc

Additional context

No response

[lmatz]

In both "create table stuck" cases, the chaos we ingested include

[ name: networkchaos-frontend-meta-longcmkf-20240511-030816, mode: one, action: partition, direction: from, duration: 600s ]

There is a network partition between the frontend node and the meta node for 10 minutes.

We expect the create table to either be successful or return an error message after 10 minutes.

[fuyufjh]

Check the await-tree dump file and didn't find anything wrong. The streaming was running normally. I guess the problem was because no one notifies the frontend create table request to stop waiting and returns an error.

There is a network partition between the frontend node and the meta node for 10 minutes.

I think the call stack will be like

frontend request (protocol layer)
  - create table handler
     - RPC call to Meta <-- waiting here

Perhaps the RPC call to Meta didn't fail as expected. Let's dive into it.

[cyliu0]

We reran those two for v1.9.0-rc-1.

One hit this again https://buildkite.com/risingwave-test/chaos-mesh/builds/829#018f7135-e456-440d-8212-0d6b161e9dee.

Another one passed https://buildkite.com/risingwave-test/chaos-mesh/builds/830.

So the reproducible rate is quite high but not 100%