http1 clientStreamConnection goroutine leak

[rayowang]

Describe the bug

When there are many 502 or 504, it is found that the clientStreamConnection goroutine under the http1 protocol cannot end normally.

For details, you can see the goroutine stack of ppof when debug=1 and debug=2：

debug=1

goroutine profile: total 2375
1252 @ 0x14d6ae0 0x14e6ed1 0x3396f96 0x339d02a 0x1840f8f 0x1506e11
#	0x3396f95	mosn.io/mosn/pkg/stream/http.(*clientStreamConnection).serve+0xa5	/go/src/mosn.io/mosn/pkg/stream/http/stream.go:275
#	0x339d029	mosn.io/mosn/pkg/stream/http.newClientStreamConnection.func1+0x29	/go/src/mosn.io/mosn/pkg/stream/http/stream.go:267
#	0x1840f8e	mosn.io/pkg/utils.GoWithRecover.func1+0x4e				/go/src/mosn.io/mosn/vendor/mosn.io/pkg/utils/goroutine.go:58

debug=2

goroutine 32179166 [select, 131 minutes]:
mosn.io/mosn/pkg/stream/http.(*clientStreamConnection).serve(0xc43eff20b0)
	/go/src/mosn.io/mosn/pkg/stream/http/stream.go:275 +0xa6
mosn.io/mosn/pkg/stream/http.newClientStreamConnection.func1()
	/go/src/mosn.io/mosn/pkg/stream/http/stream.go:267 +0x2a
mosn.io/pkg/utils.GoWithRecover.func1(0x0, 0xc43efe1180)
	/go/src/mosn.io/mosn/vendor/mosn.io/pkg/utils/goroutine.go:58 +0x4f
created by mosn.io/pkg/utils.GoWithRecover
	/go/src/mosn.io/mosn/vendor/mosn.io/pkg/utils/goroutine.go:42 +0x49

goroutine 36572737 [select, 27 minutes]:
mosn.io/mosn/pkg/stream/http.(*clientStreamConnection).serve(0xc3ea355d90)
	/go/src/mosn.io/mosn/pkg/stream/http/stream.go:275 +0xa6
mosn.io/mosn/pkg/stream/http.newClientStreamConnection.func1()
	/go/src/mosn.io/mosn/pkg/stream/http/stream.go:267 +0x2a
mosn.io/pkg/utils.GoWithRecover.func1(0x0, 0xc4fdb83900)
	/go/src/mosn.io/mosn/vendor/mosn.io/pkg/utils/goroutine.go:58 +0x4f
created by mosn.io/pkg/utils.GoWithRecover
	/go/src/mosn.io/mosn/vendor/mosn.io/pkg/utils/goroutine.go:42 +0x49

[alpha-baby]

https://github.com/mosn/mosn/blob/master/pkg/stream/http/stream.go#L712-L728

看起来是这里 s.connection.requestSent <- true 没有被调用

这个看起来是有点问题 @taoyuanyuan

[rayowang]

@alpha-baby 我看这个doSend函数并不会返回任何error，除非异常退出了https://github.com/mosn/mosn/blob/master/pkg/stream/http/stream.go#L743C1-L746C2
而且没有 [stream] [http] send client request error
主要都是 [stream] [http] send server response error