Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(server): near-duplicate detection #8228

Merged
merged 49 commits into from
May 16, 2024
Merged

feat(server): near-duplicate detection #8228

merged 49 commits into from
May 16, 2024

Commits on May 16, 2024

  1. duplicate detection job, entity, config

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    2d5d21a View commit details
    Browse the repository at this point in the history

  2. queueing

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    1a3fbb5 View commit details
    Browse the repository at this point in the history

  3. job panel, update api

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    77a67b9 View commit details
    Browse the repository at this point in the history

  4. use embedding in db instead of fetching

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    3d78182 View commit details
    Browse the repository at this point in the history

  5. disable concurrency

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    3b29523 View commit details
    Browse the repository at this point in the history

  6. only queue visible assets

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    e988110 View commit details
    Browse the repository at this point in the history

  7. handle multiple duplicateIds

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    0b7b0d2 View commit details
    Browse the repository at this point in the history

  8. update concurrent queue check

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    6adf199 View commit details
    Browse the repository at this point in the history

  9. add provider

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    540c7a2 View commit details
    Browse the repository at this point in the history

  10. add web placeholder, server endpoint, migration, various fixes

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    dbdc113 View commit details
    Browse the repository at this point in the history

  11. update sql

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    ad70903 View commit details
    Browse the repository at this point in the history

  12. select embedding by default

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    7decbb3 View commit details
    Browse the repository at this point in the history

  13. rename variable

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    fd88cfb View commit details
    Browse the repository at this point in the history

  14. simplify

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    824ad11 View commit details
    Browse the repository at this point in the history

  15. remove separate entity, handle re-running with different threshold, s… 

    …et default back to 0.02
    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    421e1d4 View commit details
    Browse the repository at this point in the history

  16. fix tests

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    b0377b8 View commit details
    Browse the repository at this point in the history

  17. add tests

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    9a3dc52 View commit details
    Browse the repository at this point in the history

  18. add index to entity

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    5369430 View commit details
    Browse the repository at this point in the history

  19. formatting

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    b238f61 View commit details
    Browse the repository at this point in the history

  20. update asset mock

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    f26c71f View commit details
    Browse the repository at this point in the history

  21. fix upsertJobStatus signature

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    f8eacd8 View commit details
    Browse the repository at this point in the history

  22. update sql

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    ef75fa6 View commit details
    Browse the repository at this point in the history

  23. formatting

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    8c0d071 View commit details
    Browse the repository at this point in the history

  24. default to 0.03

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    f39d061 View commit details
    Browse the repository at this point in the history

  25. optimize clustering

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    965f1bd View commit details
    Browse the repository at this point in the history

  26. use asset's duplicateId if present

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    f217e63 View commit details
    Browse the repository at this point in the history

  27. update sql

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    be9a678 View commit details
    Browse the repository at this point in the history

  28. update tests

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    ec31d64 View commit details
    Browse the repository at this point in the history

  29. expose admin setting

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    052fb2f View commit details
    Browse the repository at this point in the history

  30. refactor

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    1baa2fa View commit details
    Browse the repository at this point in the history

  31. formatting

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    6370428 View commit details
    Browse the repository at this point in the history

  32. skip if ml is disabled

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    185c085 View commit details
    Browse the repository at this point in the history

  33. debug trash e2e

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    846fb13 View commit details
    Browse the repository at this point in the history

  34. remove from web

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    0a06662 View commit details
    Browse the repository at this point in the history

  35. remove from sidebar

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    3b40054 View commit details
    Browse the repository at this point in the history

  36. test if ml is disabled

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    6284ec7 View commit details
    Browse the repository at this point in the history

  37. update sql

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    afaf290 View commit details
    Browse the repository at this point in the history

  38. separate duplicate detection from clip in config, disable by default … 

    …for now
    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    65b4ed2 View commit details
    Browse the repository at this point in the history

  39. fix doc

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    e8aa664 View commit details
    Browse the repository at this point in the history

  40. lower minimum maxDistance

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    9c07657 View commit details
    Browse the repository at this point in the history

  41. update api

    @mertalev
    mertalev authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    73cc57c View commit details
    Browse the repository at this point in the history

  42. Add and Use Duplicate Detection Feature Flag (#9364) 

    * Add Duplicate Detection Flag

* Use Duplicate Detection Flag

* Attempt Fixes for Failing Checks

* lower minimum `maxDistance`

* fix tests

---------

Co-authored-by: mertalev <101130780+mertalev@users.noreply.github.com>
    @NicholasFlamy @mertalev
    2 people authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    39e5069 View commit details
    Browse the repository at this point in the history

  43. chore: fixes and additions after rebase

    @zackpollard
    zackpollard authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    5160ce5 View commit details
    Browse the repository at this point in the history

  44. chore: update api (remove new Role enum)

    @zackpollard
    zackpollard authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    3167d5c View commit details
    Browse the repository at this point in the history

  45. fix: left join smart search so getAll works without machine learning

    @zackpollard
    zackpollard authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    5e46b7c View commit details
    Browse the repository at this point in the history

  46. test: trash e2e go back to checking length of assets is zero

    @zackpollard
    zackpollard authored and Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    a5dbef9 View commit details
    Browse the repository at this point in the history

  47. chore: regen api after rebase

    Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    d3eb872 View commit details
    Browse the repository at this point in the history

  48. test: fix tests after rebase

    Zack Pollard committed May 16, 2024
    Configuration menu
    Copy the full SHA
    28a6e62 View commit details
    Browse the repository at this point in the history

  49. redundant join

    @mertalev
    mertalev committed May 16, 2024
    Configuration menu
    Copy the full SHA
    95eac75 View commit details
    Browse the repository at this point in the history