dfdaemon channel keeps re-creating

[heybronson]

Bug report:

all dfdaemon pods continuously create/delete channels indefinitely - channel number starts at #1 and is now at #174321 in the below logs.

dragonfly-dfdaemon-44dqr dfdaemon 2024-02-26T19:39:57.108Z      INFO    grpclog/grpclog.go:37   [core][Channel #174106] Channel switches to new LB policy "pick_first"
dragonfly-dfdaemon-44dqr dfdaemon 2024-02-26T19:39:57.108Z      INFO    grpclog/grpclog.go:37   [core][Channel #174106 SubChannel #174107] Subchannel created
dragonfly-dfdaemon-44dqr dfdaemon 2024-02-26T19:39:57.108Z      INFO    grpclog/grpclog.go:37   [core][Channel #174106] Channel Connectivity change to CONNECTING
dragonfly-dfdaemon-44dqr dfdaemon 2024-02-26T19:39:57.108Z      INFO    grpclog/grpclog.go:37   [core][Channel #174106] Channel exiting idle mode
dragonfly-dfdaemon-44dqr dfdaemon 2024-02-26T19:39:57.108Z      INFO    grpclog/grpclog.go:37   [core][Channel #174106 SubChannel #174107] Subchannel Connectivity change to CONNECTING
dragonfly-dfdaemon-44dqr dfdaemon 2024-02-26T19:39:57.108Z      INFO    grpclog/grpclog.go:37   [core][Channel #174106 SubChannel #174107] Subchannel picks a new address "dragonfly-scheduler.infra.svc.cluster.local:8002" to connect
dragonfly-dfdaemon-44dqr dfdaemon 2024-02-26T19:39:57.112Z      INFO    grpclog/grpclog.go:37   [core][Channel #174106 SubChannel #174107] Subchannel Connectivity change to READY
dragonfly-dfdaemon-44dqr dfdaemon 2024-02-26T19:39:57.112Z      INFO    grpclog/grpclog.go:37   [core][Channel #174106] Channel Connectivity change to READY
dragonfly-dfdaemon-44dqr dfdaemon 2024-02-26T19:39:57.113Z      INFO    grpclog/grpclog.go:37   [core][Channel #174106] Channel Connectivity change to SHUTDOWN
dragonfly-dfdaemon-44dqr dfdaemon 2024-02-26T19:39:57.113Z      INFO    grpclog/grpclog.go:37   [core][Channel #174106] Closing the name resolver
dragonfly-dfdaemon-44dqr dfdaemon 2024-02-26T19:39:57.113Z      INFO    grpclog/grpclog.go:37   [core][Channel #174106] ccBalancerWrapper: closing
dragonfly-dfdaemon-44dqr dfdaemon 2024-02-26T19:39:57.113Z      INFO    grpclog/grpclog.go:37   [core][Channel #174106 SubChannel #174107] Subchannel Connectivity change to SHUTDOWN
dragonfly-dfdaemon-44dqr dfdaemon 2024-02-26T19:39:57.113Z      INFO    grpclog/grpclog.go:37   [core][Channel #174106 SubChannel #174107] Subchannel deleted
dragonfly-dfdaemon-44dqr dfdaemon 2024-02-26T19:39:57.113Z      INFO    grpclog/grpclog.go:37   [core][Channel #174106] Channel deleted

How can we resolve this?

Expected behavior:

dfdaemon not continuously creating/deleting channels.

Environment:

• Dragonfly version: 2.1.31
• OS: AL2
• Kernel (e.g. uname -a): 5.10.209-198.812.amzn2.x86_64