refactor(userSchema): unique index definitions using partialFilterExpression instead of sparse

[danny-avila]

Addresses CosmosDB compatibility issue

[strayer]

I'm having a bit of trouble testing this. I checked out the branch and created a serverless RU-based CosmosDB for MongoDB on Azure (4.2).

When running the api container this happens:

❯ docker compose -f my-compose.yaml up --force-recreate api
[+] Running 1/0
 ✔ Container librechat-cosmosdbtest-2-api-1  Recreated                                                                                                                                                      0.1s
Attaching to api-1
api-1  |
api-1  | > LibreChat@0.6.9 backend
api-1  | > cross-env NODE_ENV=production node api/server/index.js
api-1  |
api-1  | 2024-02-19 08:48:02 info: [Optional] Redis not initialized. Note: Redis support is experimental.
api-1  | 2024-02-19 08:48:03 info: Connected to MongoDB
api-1  | 2024-02-19 08:48:05 error: There was an uncaught error: a collection 'LibreChat.logs' already exists
api-1 exited with code 1

Running it again shows this:

❯ docker compose -f my-compose.yaml up --force-recreate api
[+] Running 1/0
 ✔ Container librechat-cosmosdbtest-2-api-1  Recreated                                                                                                                                                      0.0s
Attaching to api-1
api-1  |
api-1  | > LibreChat@0.6.9 backend
api-1  | > cross-env NODE_ENV=production node api/server/index.js
api-1  |
api-1  | 2024-02-19 08:48:32 info: [Optional] Redis not initialized. Note: Redis support is experimental.
api-1  | 2024-02-19 08:48:33 info: Connected to MongoDB
api-1  | 2024-02-19 08:48:33 error: There was an uncaught error: Error=13, Details='Response status code does not indicate success: Forbidden (403); Substatus: 0; ActivityId: 6b5b046e-77e0-4798-b737-8a9a1ba1089e; Reason: (Message: {"Errors":["The unique index cannot be modified. To change the unique index, remove the collection and re-create a new one."]}
api-1  | ActivityId: 6b5b046e-77e0-4798-b737-8a9a1ba1089e, Request URI: /apps/3ff79e64-935b-4fa1-bba6-1ec7a0023582/services/8606c4a4-8694-49f9-a212-14e5f2fb0433/partitions/dd1db87c-77cf-43d8-8e8a-94682b9339bd/replicas/133453239069532763p, RequestStats:
api-1  | RequestStartTime: 2024-02-19T08:48:33.8531873Z, RequestEndTime: 2024-02-19T08:48:33.8610585Z,  Number of regions attempted:1
api-1  | {"systemHistory":[{"dateUtc":"2024-02-19T08:47:42.7050197Z","cpu":1.978,"memory":473539156.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0719,"availableThreads":32764,"minThreads":48,"maxThreads":32767},"numberOfOpenTcpConnection":3574},{"dateUtc":"2024-02-19T08:47:52.7156998Z","cpu":1.151,"memory":473475100.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1417,"availableThreads":32764,"minThreads":48,"maxThreads":32767},"numberOfOpenTcpConnection":3539},{"dateUtc":"2024-02-19T08:48:02.7261881Z","cpu":0.605,"memory":473457576.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0915,"availableThreads":32764,"minThreads":48,"maxThreads":32767},"numberOfOpenTcpConnection":3541},{"dateUtc":"2024-02-19T08:48:12.7367860Z","cpu":1.174,"memory":473480820.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0776,"availableThreads":32764,"minThreads":48,"maxThreads":32767},"numberOfOpenTcpConnection":3549},{"dateUtc":"2024-02-19T08:48:22.7474597Z","cpu":1.905,"memory":473846128.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0682,"availableThreads":32764,"minThreads":48,"maxThreads":32767},"numberOfOpenTcpConnection":3501},{"dateUtc":"2024-02-19T08:48:32.7580479Z","cpu":1.437,"memory":473858636.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0811,"availableThreads":32764,"minThreads":48,"maxThreads":32767},"numberOfOpenTcpConnection":3511}]}
api-1  | RequestStart: 2024-02-19T08:48:33.8532173Z; ResponseTime: 2024-02-19T08:48:33.8610585Z; StoreResult: StorePhysicalAddress: rntbd://10.0.0.19:11300/apps/3ff79e64-935b-4fa1-bba6-1ec7a0023582/services/8606c4a4-8694-49f9-a212-14e5f2fb0433/partitions/dd1db87c-77cf-43d8-8e8a-94682b9339bd/replicas/133453239069532763p, LSN: 181, GlobalCommittedLsn: 181, PartitionKeyRangeId: , IsValid: True, StatusCode: 403, SubStatusCode: 0, RequestCharge: 1.57, ItemLSN: -1, SessionToken: -1#181, UsingLocalLSN: False, TransportException: null, BELatencyMs: 6.75, ActivityId: 6b5b046e-77e0-4798-b737-8a9a1ba1089e, RetryAfterInMs: , ReplicaHealthStatuses: [(port: 11300 | status: Connected | lkt: 2/19/2024 8:48:33 AM)], TransportRequestTimeline: {"requestTimeline":[{"event": "Created", "startTimeUtc": "2024-02-19T08:48:33.8532183Z", "durationInMs": 0.01},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2024-02-19T08:48:33.8532283Z", "durationInMs": 0.0025},{"event": "Pipelined", "startTimeUtc": "2024-02-19T08:48:33.8532308Z", "durationInMs": 0.0922},{"event": "Transit Time", "startTimeUtc": "2024-02-19T08:48:33.8533230Z", "durationInMs": 7.4728},{"event": "Received", "startTimeUtc": "2024-02-19T08:48:33.8607958Z", "durationInMs": 0.0873},{"event": "Completed", "startTimeUtc": "2024-02-19T08:48:33.8608831Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0,"lastSendAttempt":"2024-02-19T08:48:33.8503062Z","lastSend":"2024-02-19T08:48:33.8503438Z","lastReceive":"2024-02-19T08:48:33.8510444Z"},"requestSizeInBytes":2277,"requestBodySizeInBytes":1746,"responseMetadataSizeInBytes":172,"responseBodySizeInBytes":126};
api-1  |  ResourceType: Collection, OperationType: Replace
api-1  | , SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, Windows/10.0.20348 cosmos-netstandard-sdk/3.18.0);
api-1 exited with code 1

Weirdly enough the same happens on main branch. I don't think I'm doing something wrong here, but I'm also a bit confused about what is happening. When starting the container the CosmosDB is definitely empty. After starting the container some schemas exist:

[strayer]

Some progress: I managed to resolve the duplicate collection error by commenting out the keyVMongo object creation here: https://github.com/danny-avila/LibreChat/blob/partial-filter-index/api/cache/keyvMongo.js#L6

It looks like something goes wrong when instantiating that class that breaks migrations or something? I added a redis cache to avoid MongoDB here and both errors from my previous comment disappeared. I was able to create two users successfully.

cosmosuser@SandboxHost-638439284691316803:~$ MongoDB shell version v3.6.8
connecting to: mongodb://foobar.mongo.cosmos.azure.com:10255/
MongoDB server version: 4.2.0
WARNING: shell and server versions do not match
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
        http://docs.mongodb.org/
Questions? Try the support group
        http://groups.google.com/group/mongodb-user
globaldb:PRIMARY> use test;
switched to db test
globaldb:PRIMARY> db.users.find()
{ "_id" : ObjectId("65d320e2702a7e9cf2c04ae2"), "name" : "test", "username" : "test", "email" : "test@test.de", "emailVerified" : false, "password" : "$2a$10$DJGHCCUUvDAESPKeDD7JFeT4IKLVBkwOAuMcLSxIweD2QAAD/sraO", "avatar" : null, "provider" : "local", "role" : "ADMIN", "plugins" : [ ], "refreshToken" : [ ], "createdAt" : ISODate("2024-02-19T09:35:30.799Z"), "updatedAt" : ISODate("2024-02-19T09:35:30.799Z"), "__v" : 0 }
{ "_id" : ObjectId("65d320ff702a7e9cf2c04b54"), "name" : "test2", "username" : "test2", "email" : "test2@test.de", "emailVerified" : false, "password" : "$2a$10$q.2jYldTj16yAhaMdHckkeUGnqWvC7pli.Tg5bl/Gh3hd8eq9fYUa", "avatar" : null, "provider" : "local", "role" : "USER", "plugins" : [ ], "refreshToken" : [ ], "createdAt" : ISODate("2024-02-19T09:35:59.702Z"), "updatedAt" : ISODate("2024-02-19T09:35:59.702Z"), "__v" : 0 }

I'm now trying to figure out the Azure OpenID authentication to test that as well.

Edit: Looks like I'm running into this issue: #1521

Don't think I can progress until that weird MongoDB logs collection error is solved :/