Mobile clients do not receive push notifications

[fogleins]

Description:

Clients do not receive push notifications about new messages. Both Android and iOS apps show the workspace version as 6.6.0, although the admin panel reports 6.6.4. When sending a message to someone, an error occurs saying unsupported version (see logs below), and the recipient does not receive a push notification. Sometimes restarting the docker container solves the issue for a few hours, but after that, the issue comes back again. This problem started about 1-2 months ago, most likely after upgrading to 6.6.x. Notifications worked fine before that.

Steps to reproduce

1. Send a message to someone who has the mobile version of RocketChat installed

Expected behavior

Recipients should get a push notification about new messages.

Actual behavior

No push notifications are sent.

Server Setup Information:

• Version of Rocket.Chat Server: 6.6.4
• Operating System: SUSE Linux Enterprise Server 15 SP5
• Deployment Method: docker
• Number of Running Instances: 1
• NodeJS Version: 14.21.3
• MongoDB Version: 6.0.12

Client Setup Information

• Mobile App Version: 4.47.2.59390
• Operating System: Android

Relevant logs:

{"level":50,"time":"2024-03-11T16:54:42.516Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Error sending push to gateway (0 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-03-11T16:54:42.516Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Trying sending push to gateway again in 60000 'milliseconds'"}
{"level":50,"time":"2024-03-11T16:54:42.955Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Error sending push to gateway (0 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-03-11T16:54:42.955Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Trying sending push to gateway again in 60000 'milliseconds'"}
{"level":50,"time":"2024-03-11T16:54:48.103Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Error sending push to gateway (2 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-03-11T16:54:48.104Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Trying sending push to gateway again in 240000 'milliseconds'"}
{"level":50,"time":"2024-03-11T16:55:42.914Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Error sending push to gateway (1 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-03-11T16:55:42.915Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Trying sending push to gateway again in 120000 'milliseconds'"}
{"level":50,"time":"2024-03-11T16:55:43.438Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Error sending push to gateway (1 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-03-11T16:55:43.438Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Trying sending push to gateway again in 120000 'milliseconds'"}
{"level":50,"time":"2024-03-11T16:55:43.890Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Error sending push to gateway (2 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-03-11T16:55:43.891Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Trying sending push to gateway again in 240000 'milliseconds'"}
{"level":50,"time":"2024-03-11T16:55:44.257Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Error sending push to gateway (2 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-03-11T16:55:44.257Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Trying sending push to gateway again in 240000 'milliseconds'"}
{"level":50,"time":"2024-03-11T16:57:43.456Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Error sending push to gateway (2 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-03-11T16:57:43.457Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Trying sending push to gateway again in 240000 'milliseconds'"}
{"level":50,"time":"2024-03-11T16:57:43.800Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Error sending push to gateway (2 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-03-11T16:57:43.801Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Trying sending push to gateway again in 240000 'milliseconds'"}
{"level":50,"time":"2024-03-11T16:58:48.523Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Error sending push to gateway (3 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-03-11T16:58:48.523Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Trying sending push to gateway again in 480000 'milliseconds'"}
{"level":50,"time":"2024-03-11T16:59:44.295Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Error sending push to gateway (3 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-03-11T16:59:44.296Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Trying sending push to gateway again in 480000 'milliseconds'"}
{"level":50,"time":"2024-03-11T16:59:44.642Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Error sending push to gateway (3 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-03-11T16:59:44.642Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Trying sending push to gateway again in 480000 'milliseconds'"}
{"level":50,"time":"2024-03-11T17:01:43.817Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Error sending push to gateway (3 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-03-11T17:01:43.818Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Trying sending push to gateway again in 480000 'milliseconds'"}
{"level":50,"time":"2024-03-11T17:01:44.205Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Error sending push to gateway (3 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-03-11T17:01:44.206Z","pid":1,"hostname":"f62fbb11f90e","name":"Push","msg":"Trying sending push to gateway again in 480000 'milliseconds'"}

The logs are from a 6.6.1 deployment nearly a month ago, but after upgrading to 6.6.4 the issue still persists.
Is there anything I should look into to help resolve this issue?

[meenakshiiyer2531]

Can I please handle this issue?

[ChetanGoti]

Upgraded to 6.7.0 and we still see this error and no notifications for mobile clients.

[geekgonecrazy]

Can you give hitting sync a try and then trying a push again?

Unfortunately this is a service side thing.

If you have time for debugging join https://open.rocket.chat/channels/support and drop a message. I can DM you and look at your case directly to see whats happening.

Also be sure you are using a final release not a develop image right?

[ChetanGoti]

Thanks. This is fixed for us without performing any actions on our end.

We started getting the push notifications on mobile apps from yesterday. The last error we saw was at 2024-04-09T07:30:00.160Z

[fogleins]

Can you give hitting sync a try and then trying a push again?

Unfortunately this is a service side thing.

If you have time for debugging join https://open.rocket.chat/channels/support and drop a message. I can DM you and look at your case directly to see whats happening.

Also be sure you are using a final release not a develop image right?

Thanks. I've clicked the Sync button on the Subscriptions page, and mobile clients have started to receive notifications, and no errors have been logged. I'll keep an eye on the logs in the next few days, and will report back if anything comes up.
Thanks again!

[fogleins]

After about 3 hours of no problems, the errors started appearing again. It seems to me that when RocketChat tries to automatically sync the license status, the token is not read correctly:

# Previous error at 2024-04-15T08:44:15.650Z
{"level":30,"time":"2024-04-15T12:00:03.386Z","pid":1,"hostname":"85890fef3c06","name":"System","msg":"Failed to sync with Rocket.Chat Cloud","function":"announcementSync","err":{"type":"CloudWorkspaceAccessTokenEmptyError","message":"Workspace access token is empty","stack":"Error: Workspace access token is empty\n    at app/cloud/server/functions/syncWorkspace/announcementSync.ts:86:10\n    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40"}}
{"level":30,"time":"2024-04-15T12:00:03.390Z","pid":1,"hostname":"85890fef3c06","name":"System","msg":"Falling back to legacy sync","function":"syncCloudData"}
{"level":50,"time":"2024-04-15T12:00:32.917Z","pid":1,"hostname":"85890fef3c06","name":"Push","msg":"Error sending push to gateway (0 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-04-15T12:00:32.917Z","pid":1,"hostname":"85890fef3c06","name":"Push","msg":"Trying sending push to gateway again in 60000 'milliseconds'"}
{"level":50,"time":"2024-04-15T12:00:33.424Z","pid":1,"hostname":"85890fef3c06","name":"Push","msg":"Error sending push to gateway (0 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-04-15T12:00:33.426Z","pid":1,"hostname":"85890fef3c06","name":"Push","msg":"Trying sending push to gateway again in 60000 'milliseconds'"}
{"level":50,"time":"2024-04-15T12:00:58.022Z","pid":1,"hostname":"85890fef3c06","name":"Push","msg":"Error sending push to gateway (0 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-04-15T12:00:58.023Z","pid":1,"hostname":"85890fef3c06","name":"Push","msg":"Trying sending push to gateway again in 60000 'milliseconds'"}
{"level":50,"time":"2024-04-15T12:00:58.448Z","pid":1,"hostname":"85890fef3c06","name":"Push","msg":"Error sending push to gateway (0 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-04-15T12:00:58.450Z","pid":1,"hostname":"85890fef3c06","name":"Push","msg":"Trying sending push to gateway again in 60000 'milliseconds'"}
{"level":50,"time":"2024-04-15T12:01:33.313Z","pid":1,"hostname":"85890fef3c06","name":"Push","msg":"Error sending push to gateway (1 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-04-15T12:01:33.314Z","pid":1,"hostname":"85890fef3c06","name":"Push","msg":"Trying sending push to gateway again in 120000 'milliseconds'"}
{"level":50,"time":"2024-04-15T12:01:33.841Z","pid":1,"hostname":"85890fef3c06","name":"Push","msg":"Error sending push to gateway (1 try) ->","err":"{\"errorMsg\":\"unsupported version\"}"}
{"level":30,"time":"2024-04-15T12:01:33.841Z","pid":1,"hostname":"85890fef3c06","name":"Push","msg":"Trying sending push to gateway again in 120000 'milliseconds'"}

This time, manually syncing worked only after the third try (the sync succeeded according to the toast on the admin panel, but notification still didn't work).

[Simon-Design]

same issue for me... manually syncing works but only for a few hours

[reetp]

Both Android and iOS apps show the workspace version as 6.6.0, although the admin panel reports 6.6.4.

Apparently for 'security'. Personally I think that if they are dropping the last digit it should go entirely. eg show just 6.6

As per Aarons message

If you have time for debugging join https://open.rocket.chat/channels/support and drop a message. I can DM you and look at your case directly to see whats happening.

This isn't a bug with Rocket per se but backend services I believe.

Please pick this up on open.