Add cluster option to enable "upmap-read" in the balancer module

[travisn]

Is this a bug report or feature request?

• Feature Request

What should the feature do:
With Ceph Squid, there is a new optimization for the balancer to balance reads as well as writes. The default option "upmap" only balances writes. Now with the new option "upmap-read", both reads and writes can be optimized.

Rook would need to run this command:

ceph balancer mode upmap-read

To give flexibility for module settings that could be easily expanded in the future for other module settings, this could be set in the cluster CR such as:

  mgr:
    modules:
      - name: balancer
        enabled: true
        settings:
          mode: upmap-read

What is use case behind this feature:
Enable more efficient reads.

The workaround is to run that balancer command in the toolbox.

[travisn]

For more details on the Ceph feature, see ceph/ceph#53988

[subhamkrai]

Assigning it to you @sp98 since he is already looking at this.