[Bug] metrics alloc panics after hot upgrade

[dengqian]

2023-06-05 16:19:35,506 goroutine panic: runtime error: index out of range [4294967295] with length 161318
goroutine 13716 [running]:
runtime/debug.Stack()
    /tmp/usr/local/go/src/runtime/debug/stack.go:24 +0x65
gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/pkg/utils.defaultRecoverLogger({0x2055240, 0xc0001b4010}, {0x1c5c940, 0xc00235c018})
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/pkg/utils/goroutine.go:37 +0x5a
gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/pkg/utils.GoWithRecover.func1.1()
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/pkg/utils/goroutine.go:45 +0x62
panic({0x1c5c940, 0xc00235c018})
    /tmp/usr/local/go/src/runtime/panic.go:1038 +0x215
gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/metrics/shm.(*hashSet).Alloc(0xc001350700, {0xc001db6180, 0xc000cbeed0})
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/metrics/shm/hashset.go:167 +0x2e5
gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/metrics/shm.(*zone).alloc(0xc001344620, {0xc001db6180, 0x31})
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/metrics/shm/zone.go:170 +0x8a
gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/metrics/shm.NewShmCounterFunc.func1()
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/metrics/shm/counter.go:59 +0x6d
reflect.Value.call({0x1a10880, 0xc001f1a588, 0x13}, {0x1db3f41, 0x4}, {0x0, 0x0, 0x0})
    /tmp/usr/local/go/src/reflect/value.go:556 +0x845
reflect.Value.Call({0x1a10880, 0xc001f1a588, 0x1dc700a}, {0x0, 0x0, 0x0})
    /tmp/usr/local/go/src/reflect/value.go:339 +0xc5
gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/github.com/rcrowley/go-metrics.(*StandardRegistry).GetOrRegister(0xc001a1f400, {0x1dc700a, 0x10}, {0x1a10880, 0xc001f1a588})
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/github.com/rcrowley/go-metrics/registry.go:101 +0x265
gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/metrics.(*metrics).Counter.func1()
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/metrics/store.go:156 +0x75
gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/metrics.(*metrics).Counter(0xc00167aaf0, {0x1dc700a, 0x1dbd2f5})
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/metrics/store.go:162 +0x116
gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/proxy.newStats({0x20bae18, 0xc00167aaf0})
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/proxy/stats.go:67 +0x4e
gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/proxy.newListenerStats({0xc0004dcb30, 0xc})
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/proxy/stats.go:58 +0x8c
gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/proxy.NewProxy({0x209d3c8, 0xc001f72180}, 0xc00167aa80)
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/proxy/proxy.go:135 +0x3bf
gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/filter/network/proxy.(*genericProxyFilterConfigFactory).CreateFilterChain(0xc001a7b0b0, {0x209d3c8, 0xc001f72180}, {0x7f2254e1aad0, 0xc0021f6000})
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/filter/network/proxy/factory.go:55 +0xc5
gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/server.(*activeListener).OnNewConnection(0xc0010b80c0, {0x209d3c8, 0xc001f72180}, {0x20ddd98, 0xc00036e400})
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/server/handler.go:525 +0x5f6
gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/server.(*activeListener).newConnection(0xc0010b80c0, {0x209d3c8, 0xc001f72180}, {0x20bb130, 0xc0001b40b0})
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/server/handler.go:671 +0x2e5
gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/server.(*activeRawConn).ContinueFilterChain(0xc000f3c000, {0x209d3c8, 0xc001f72180}, 0x40)
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/server/handler.go:778 +0xb9
gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/server.(*activeListener).OnAccept(0xc0010b80c0, {0x20bb130, 0xc0001b40b0}, 0x0, {0x0, 0x0}, 0x0, {0x0, 0x0, 0x0}, ...)
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/server/handler.go:518 +0x9e5
gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/network.(*listener).accept.func1()
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/mosn/pkg/network/listener.go:428 +0x90
gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/pkg/utils.GoWithRecover.func1()
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/pkg/utils/goroutine.go:58 +0x5b
created by gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/pkg/utils.GoWithRecover
    /home/admin/63_20230327172516619_476609323_code/rpm_workspace/go/src/gitlab.alibaba-inc.com/apsara-edge/mosn/vendor/mosn.io/pkg/utils/goroutine.go:42 +0x77

This panic accidentally occurs after new mosn hot upgrade started. This panic is fatal because all requests will fail. I think it may be caused by the reuse of the shm file, two process write the same file cause data exception.

[doujiang24]

I think it may be caused by the reuse of the shm file, two process write the same file cause data exception.

If it is, seems it's hard to fix it? maybe we need to disable metrics for old mosn? during hot upgrade.