Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

enhance documentation with package upgrade description #1707

Open
tomatolog opened this issue Jan 4, 2024 · 2 comments · May be fixed by #1729
Open

enhance documentation with package upgrade description #1707

tomatolog opened this issue Jan 4, 2024 · 2 comments · May be fixed by #1729
Assignees
@sanikolaev
Labels
enhancement

Comments

@tomatolog
Copy link
Contributor

we have only instruction on ManticoreSearch installation at the manual but not about package upgrade. The only mention Changelog#Upgrade is

In case you run Manticoresearch in a distributed environment with multiple instances make sure your first upgrade agents, then the masters.

  1. however it is not clear why agents should be upgraded first (as they could receive request with the old API version from master then reply with the same version as request received) but sending the request with the new API version cause the old agent to reply with the error.
  2. There is no mention of how to upgrade from the very old versions - to upgrade only one agent and make sure it receives and replies well or check Changelog about breaking change in the API protocol in case of the distributed index.
  3. That node with the indexer should be upgraded the last to make indexes these could be loaded by the old and the new daemons.
  4. Need to keep separate RT and plain indexes from the upgraded daemon and old daemon as old daemon can not load indexes of the new versions, ie can not restore backup of the new version indexes into installation with the old daemon.
  5. Need to keep in mind all previous points while upgrading nodes at the cluster. As replication protocol version along with the index version could be changed in the new package. This way if upgraded node would be a donor it breaks the joiner node with the old daemon by the new index version. As there is no check on donor selection that version matches.
    It could be better to join the upgraded node with the Galera option gmcast.segment=20 to make sure the new nodes will form the new segment and do not SST the new indexes into the old nodes.
    Or upgrade cluster with full shutdown and upgrade all nodes at the same time.
@sanikolaev sanikolaev linked a pull request Jan 15, 2024 that will close this issue
manual: info on upgrade #1729
Open
@sanikolaev
Copy link
Collaborator

I've prepared this PR #1729

Can you please elaborate more on the gmcast.segment=20 approach: how can it be done in Manticore Search? What commands on what nodes and in what order should be executed?

@tomatolog
Copy link
Contributor Author

The idea of the point 5 is to make separation of the old and new nodes at the cluster. That could be done by assign another gmcast.segment value for the new nodes.
I was sure that command

SET CLUSTER a GLOBAL 'gmcast.segment' = 20;

does that. However after test I see it returns error that gmcast.segment can not be changed in runtime. That means the node needs to join with this option set. However as we do not have command that remove node from cluster now such scenario looks not very friendly due to many manual steps:

  • user should stop the node
  • remove cluster property from the manticore.json to prevent node rejoin the cluster on start
  • upgrade daemon package
  • start the node
  • after node started issue cluster join with the Galera options
join cluster CLUSTER_NAME at 'ADDRESS:PORT' 'gmcast.segment=20' as options

this way the new nodes will form another segment at the cluster and will not be selected as donor for old nodes and will not push index with the new format into the old daemons.

After all nodes got upgraded it could be better to remove gmcast.segment option to reduce possible clutter in the cluster management. That also could be done now only after node got stopped and manually edit manticore.json deleting the options property.

Maybe we should not to mention point 5 and suggest to upgrade the cluster via full cluster restart described at our manual

@tomatolog tomatolog assigned sanikolaev and unassigned tomatolog Jan 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sanikolaev sanikolaev

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

manual: info on upgrade manticoresoftware/manticoresearch
2 participants
@tomatolog @sanikolaev