feat(server): near-duplicate detection (#8228)

* duplicate detection job, entity, config

* queueing

* job panel, update api

* use embedding in db instead of fetching

* disable concurrency

* only queue visible assets

* handle multiple duplicateIds

* update concurrent queue check

* add provider

* add web placeholder, server endpoint, migration, various fixes

* update sql

* select embedding by default

* rename variable

* simplify

* remove separate entity, handle re-running with different threshold, set default back to 0.02

* fix tests

* add tests

* add index to entity

* formatting

* update asset mock

* fix `upsertJobStatus` signature

* update sql

* formatting

* default to 0.03

* optimize clustering

* use asset's `duplicateId` if present

* update sql

* update tests

* expose admin setting

* refactor

* formatting

* skip if ml is disabled

* debug trash e2e

* remove from web

* remove from sidebar

* test if ml is disabled

* update sql

* separate duplicate detection from clip in config, disable by default for now

* fix doc

* lower minimum `maxDistance`

* update api

* Add and Use Duplicate Detection Feature Flag (#9364)

* Add Duplicate Detection Flag

* Use Duplicate Detection Flag

* Attempt Fixes for Failing Checks

* lower minimum `maxDistance`

* fix tests

---------

Co-authored-by: mertalev <101130780+mertalev@users.noreply.github.com>

* chore: fixes and additions after rebase

* chore: update api (remove new Role enum)

* fix: left join smart search so getAll works without machine learning

* test: trash e2e go back to checking length of assets is zero

* chore: regen api after rebase

* test: fix tests after rebase

* redundant join

---------

Co-authored-by: Nicholas Flamy <30300649+NicholasFlamy@users.noreply.github.com>
Co-authored-by: Zack Pollard <zackpollard@ymail.com>
Co-authored-by: Zack Pollard <zack@futo.org>

docs/docs/install/config-file.md

@@ -77,6 +77,10 @@ The default configuration looks like this:
       "enabled": true,
       "modelName": "ViT-B-32__openai"
     },
+    "duplicateDetection": {
+      "enabled": false,
+      "maxDistance": 0.03
+    },
     "facialRecognition": {
       "enabled": true,
       "modelName": "buffalo_l",

e2e/src/api/specs/server-info.e2e-spec.ts

@@ -66,6 +66,7 @@ describe('/server-info', () => {
       expect(body).toEqual({
         smartSearch: false,
         configFile: false,
+        duplicateDetection: false,
         facialRecognition: false,
         map: true,
         reverseGeocoding: true,

e2e/src/api/specs/trash.e2e-spec.ts

@@ -32,8 +32,7 @@ describe('/trash', () => {
       await utils.deleteAssets(admin.accessToken, [assetId]);
 
       const before = await getAllAssets({}, { headers: asBearerAuth(admin.accessToken) });
-
-      expect(before.length).toBeGreaterThanOrEqual(1);
+      expect(before).toStrictEqual([expect.objectContaining({ id: assetId, isTrashed: true })]);
 
       const { status } = await request(app).post('/trash/empty').set('Authorization', `Bearer ${admin.accessToken}`);
       expect(status).toBe(204);
@@ -57,14 +56,14 @@ describe('/trash', () => {
       const { id: assetId } = await utils.createAsset(admin.accessToken);
       await utils.deleteAssets(admin.accessToken, [assetId]);
 
-      const before = await utils.getAssetInfo(admin.accessToken, assetId);
-      expect(before.isTrashed).toBe(true);
+      const before = await getAllAssets({}, { headers: asBearerAuth(admin.accessToken) });
+      expect(before).toStrictEqual([expect.objectContaining({ id: assetId, isTrashed: true })]);
 
       const { status } = await request(app).post('/trash/restore').set('Authorization', `Bearer ${admin.accessToken}`);
       expect(status).toBe(204);
 
-      const after = await utils.getAssetInfo(admin.accessToken, assetId);
-      expect(after.isTrashed).toBe(false);
+      const after = await getAllAssets({}, { headers: asBearerAuth(admin.accessToken) });
+      expect(after).toStrictEqual([expect.objectContaining({ id: assetId, isTrashed: false })]);
     });
   });