[Feature] Facial Recognition Enhancements

[Quafley]

(Parent discussion for FR enhancements)

Summary:

• 1. Ability to show/hide faces (similar to google photos) (feat(server,web): hide faces #3262)
• 2. Option to select an alternative person thumbnail (similar to google photos) (feat(web/server): Face thumbnail selection #3081)
• 3. Explore having a person "preview" when view people in a shared album, link, partner timeline, etc.
• 4a. Improve performance of viewing all people
• 4b. Improve performance of viewing all assets for a person (feat(web): use time buckets of person detail page (3) #3557)
• 5. Add the option to link/merge two people into one (feat(web/server): merge faces #3121)
• 6. Add a confirmation modal before running recognize faces for all assets (feat: confirm before deleting all faces and people #2496)
• 7. Add FR related features into the mobile app
• 8. Possibility of a dedicated People sidebar item (feat(web): people sidebar link #4257)
• 9. Explore reducing required clicks to tag people (feat(web): add better face management UI action #3328)
• 10. Add the option to remove/update a wrongly tagged person (feat: re-assign person faces (2) #4949)
• 11. Back arrow is broken when navigating to a person from the View All (people) page. (fix(web): back button on person page #2566)
• 12. Run the FR pipeline again for a specific image.

(Original Post Below)

The feature

Hi,

Awesome that the FR is implemented, it works awesome! After making use of the UI if felt a bit clunky and could perhaps use a bit of work. Down below I have listed a few things that I've noticed.

1. Currently within the UI there are 12 spots, which I presume are reserved for people with the most pictures. Although, I doubt this a bit. There are various things that came to mind when I saw this. It would be a fantastic feature to favorite certain people, this could be put under the Explore tab. At the same time adding a separate tab for Facial recognition, or People, could also be a possibility.
2. It would be nice if there is a possibility to add more than 12 people within the people tag within explore.
   
3. When you want to assign a name to someone that is not within the Explore tab you have to click on view all, then click on the person, and then you're able to add the name. A few things came to mind, add the possibility to add names to the image it self, as per shown below. Second
   
4. When clicking on a photo, the faces that are within the photo are displayed. I don't know if this is possible or even feasible, but it would be nice if the faces are displayed that are actually within the picture, instead of the general image which is associated with the person. This can help with improper matching, although I have not yet encountered this.

If there is more, I will edit it below.

EDIT:
13. When clicking on view all and selecting a person, after clicking on the back arrow in the top left. It takes you back to the explore page, resulting in needing to click on view all again. When clicking on back arrow within the browser, it does not do this. Although, I have also noticed this not working 100% of the time. This just adds another click.
14. The possibility to remove faces or hide them.
15. The possibility to unmark someone if they are incorrectly tagged.
16. A rescan on a per image basis if facial recognition does not recognize a particular face in the image. (I've noticed this happening with people who have a mouctache)

Platform

• Server
• Web
• Mobile

[jrasm91]

Thanks for documenting these!

[colin715]

I would appreciate a separate page/menu icon for People

[jrasm91]

We're pretty much sticking to the layout that Google uses right now, although we could certainly consider alternatives.

[dominikheinz]

I agree, a dedicated tab to manage everything related to people and FR would be nice.

[jrasm91]

An option to enable a dedicated "People" page link (number 8 in the list) has been added and will be in the next release.

[dominikheinz]

Great list. Another feature that came to mind after experimenting with the FR feature is the option to "link" people. For example, I have two pictures of myself, clearly in frame, but they are detected as different people. (Although to the human eye they look very similar).

So, it would be great if I could merge these "two" people into one, or link them.

I feel like a dedicated section in the settings for everything related to people and face recognition would be a good idea.

[Belphemur]

Link or merge them. I think they should be one "person".
One easy way I saw it done was to merge the people when we use the exact same name.

[bjrne]

I'd prefer same names to not be merged automatically, otherwise you have to keep in mind who you tagged already if you're not using family names...

[bread22]

"merge" is a great feature to have, my son shows up as 3 different people...

[SiriusBlue5120]

Another feature I can suggest is having the ability to set a 'Key' photo of a person that would then be the image displayed in most of the UI except where 4 above comes into play. This is possible in Apple's photos app as 'Make key photo', and in Google Photos as 'Use as feature photo'.

I think this would be great because in the current UI it is often possible for blurry or yawed faces to be key photos.

[DrSpaldo]

Some of the ideas for tweaking the UI are good ones. Having to do the view all, then click the photo, then name it, then back to the main list and start again is a little cumbersome

[Vcele]

In Google Photos there's also a way to improve recognition by manually assigning select photos to a person. Also, like in Photoprism, single faces could be moved into their own overview to prevent clutter.

[DrSpaldo]

I've just found my first person that was incorrectly linked to a people/person tag. Ie. There is 300 photos that have been tagged/linked to a particular person, however, one is incorrect and not that person. Is there a way I can unlink that photo from that person? If there is no way, can that please be added to the roadmap above?

[jrasm91]

Added!

[jeremywillans]

Is it possible to highlight in a photo where each identified face is? Scenario is when FR detects partial faces, if its possible to locate this in a photo in order to work out who it actually is to link to an existing user, or correctly identify them.

Also, the option to dismiss people/hide from the people list

[fabiannagel]

There's now a discussion for this

[Quafley]

I've added two new suggestions. To my post, as I intended the first list to be edited by the team :)

[xyxzxyz]

This is an excellent list. However, any chance you can move the additional requests above "(Original Post Below)"? I almost overlooked the #12...#15. I scrolled past Original Post and missed those additional requests.

[Quafley]

You want me to move them to the list above the post?

[Harris-KA]

My thoughts exactly! 5 would be great for me as anyone with sunglasses seems to be a different person!

[pabloscloud]

5 please :)

[henley-regatta]

I'm finding the "All People" view very slow to load, because FR has found >2000 unique faces; that's a lot of thumbnails to load.
Would it be possible to add a "Named People" and "Unknown People" (without tags) filters to the interface to help with this in everyday use?

[DrSpaldo]

I can't remember if it was in one of the Github discussions or in Discord. But I know this issue has been raised and was on the list of things to look at. I believe in part 'lazy loading' may assist with this as well..

[henley-regatta]

Thanks.
I've since discovered that the bulk of my own performance issue was due to a conflict between the way this page is being built and a Chrome extension I'm using (HoverZoom+). Time to first thumbnail with that active was >10 seconds; if I disable it it's ~1.5seconds.
(I'm writing this here mainly in case anyone else is having this sort of performance issue)

[DrSpaldo]

Very interesting @henley-regatta , I should take a look at what extensions I have running as well to see if it would help my terrible loading times and partial freezing

[radh21301]

Would it be possible to implement a feature to detect faces in videos? I know Google Photos does some magic and shows people in videos as well (not just people found in the video thumbnail). This is could be such a cool feature!

[Ziomal12]

It detects though. I even checked right now to make sure and surely when I click on one of the faces in first 3 lines of photos I've got 2 videos.

[Ziomal12]

Anyways, if anyone wants to play around with face detection in a video I'd suggest to analyze 1 frame per second of video or even less (maybe even 1 per 5 seconds)

[radh21301]

I checked again. I can see that people are detected if they are present in the video thumbnail (or in the very beginning of the video). But not when they come in the middle of the video. Maybe I am doing something wrong.

[xyxzxyz]

It does recognize faces in videos, at least in mine.

[tsemachh]

Perhaps consider allow having in the person view in video links to the positions this person appears , this would allow finding people on specific location on the video

[Smiehoo]

I have some suggestions to hide/show faces ability:

1. Merging peoples option should not show hidden faces,
2. When we want to hide more faces all faces with no name on it should show on top below named faces (it hard to find missing faces to hide when we have over 300 faces)
3. When we try to name faces and this name is already use can Immich ask to merge this two people?

[martabal]

Additionally to point 2, I think it would make more sense to be able to select multiple faces and hide them all at once whitin the faces page and not the hidden faces. Could look like this:

At this point, just use the merge editor ?

[0rn0lf]

merge editor? Where can I find it?

[martabal]

The "Merge Faces" button.

[yoyotueur]

Would it be possible to add an autocomplete when faces have not been recognised and you type for the name so you make sure you don't input the wrong format for existing people

[martabal]

Yes, this is in my TODO list

[shakyr]

The features I'd like regarding facial recognition/people:

• Ability to tag someone as in an image, disconnected from the facial recognition. I have some photos where people have their back to the camera or they're too young, but I'd still like to say they're in the photo.
• This would also be useful for tagging pets.
• Ability to blacklist photos or albums from the facial recognition. Especially if you have photos of crowded areas and a few dozen random people show up as recognised.
• Bonus points if you can blacklist all unmatched faces in a photo (so you can tag known people first and then ignore everyone else).

[AngelaDMerkel]

Google Photos absolutely has some implementation of your first feature and it is exceptionally useful. I would like to +1 for a face whitelist. Only tagging faces of those manually approved would be a good feature, especially when you may have several images of a crowded area where faces repeat, or where you have a acquaintance often in your images, but not significant enough for you to tag the face.

[Judman]

Yes, So many unknown and blurry background faces just muck up this feature.

[jhit]

Yes I already missed that feature a lot. 👍

[mraction]

It would be so nice to have this feature!

[whoindahouse]

It would also be nice to have the option to feed a face you tag back to the facial recognition model for improvement.

[jakob42]

Maybe we could add #4096 to the list, might be better here than in its own feature request.

[Cebrain]

For Documentation my report #4194
When i load the people page, i think all images will be loaded from the web server, because of that limiting factors of a proxy for example can kick in. It would be nice when the thumbnails will be loaded when the part of the page loads, instead of all thumbs

[AuraHerreroRuiz]

I would like to add, even if I'm not sure how much effort it would take, to have face recognition for pets! I would guess mainly cats and dogs, as that's what Google Photos is capable of doing, although I haven't searched if they tag other animals.

[Vyerni]

I have the very same situation. My dog entered as a person in about 4 photos, but have about 500.

[ridicolos]

Would love to see pet support too. I got two dogs with more than enough photos of both of them. Only one dog was recognized with just two photos of it

[0rn0lf]

At least your dog wasn’t recognised as a sheep 😂
But I also have a ton of animal photos and only one of them was recognised.
Also, it must be smart enough so that it doesn’t add every animal in a zoo to your people list.

[mertalev]

I'd love to add this, but there doesn't seem to be much to work with in terms of models or even training data.

You can get something similar with smart search by searching for your pet's breed / features. Combined with smart albums (once we add that), this can simulate pet detection to some extent.

[sym0nd0]

Excuse my n00b question. Would it be possible to train the local model from a selection of uploaded photos, which the user manually tags and then when a scheduled scan runs or after future uploads any that match automatically get tagged, with a yes/no vote to improve the local model?

As I've no clue what I'm talking about, I also have no idea if that's possible, a stupid idea or how difficult it would be to implement.

[aviv926]

Number 10 is done

[theshoeshiner]

Can manual tagging (#2472 (comment)) be added to the summary list? I think it's been mentioned a few different times but I don't see it called out as a separate requested feature and I don't think it falls under any of the current ones.

[Soulplayer]

Probably really difficult to implement but what I like about apple photos app is that it also suggests faces to match with a person (e.g. is this person person y?). Is this something that could be added to the list?

[xyxzxyz]

Auto add person/pet to album is a must.

[thomasloven]

Some form of "mass retagging" would be awesome too.
I have a person with 200+ assets where perhaps 100 are wrong but of the same second person.
Right now it's six clicks to retag each.

I wish I had some idea on how a better way to do it might even look...

[tippfehlr]

select all wrong faces and click on the three dots -> ‘fix incorrect match’ :)

[jakob42]

I cannot find that on v1.94.1. Where did you find that? I've got a couple pictures selected and want to change one (wrongly) recognized face into another...

[tippfehlr]

Sorry, I wasn't specific. You have to select them on the page for a face.

[search] -> people -> click on the face the images are wrongly assigned to -> select images -> click the three dots at the top right

[thomasloven]

I have no idea how I could have missed that...

Thanks a lot!

[tippfehlr]

I have two people shown as one (because of similar faces) a few times.

Currently, there is no way of manually adding faces (?), so the wrong face can’t be changed manually.
So, first step would be to allow manually creating faces
But manually making FR more specific in these cases (by telling FR which face is which) would be great.

EDIT: I just found the (much more powerful) menu when clicking on ‘fix incorrect match’ in the page for a face.

[geoleven]

One great feature would be to add the ability for facial detection / recognition to be able to run on all assets without losing the previous tags.

Let me explain:
Right now there are two options, missing and all. Missing runs for images it has not already run, all will remove all the tags (for me that I have more than 600 named people, this is a week's worth of work). Right now, and depending on some conditions (bugged out facial recognition, etc which I probably have to file as a bug at some point), I end up in a lot of situations with obvious missing faces detected and only the already named people are tagged. I would like to have a middle option to rerun the face detection on the parts of the photographs that is not tagged already. I don't really know the technical background so I am not sure this is technically easy or viable. Another take would probably be (if the previous one is undoable), to enable a per photo - on demand rerun of the facial detection. This means that although I may lose the info for the already tagged people for a photo is easy to reassign this specific one adding the new ones.

[tuupola]

Something like this is needed. It is also currently bit too easy to loose all your named persons.

[jenda69]

It would be nice to have (or is it there somewhere?) option to rescan face in a photo, similarly to "Refresh metadata" and "Refresh thumbnails" commands.
I have an external gallery where I manage my photos externally, that especially means editing RAWs. And when I'm done editing, I run a script to refresh edited output JPG photos in Immich using API command runAssetJobs with regenerate-thumbnail and refresh-metadata parameters. I get new thumbnails etc. but if I cropped a photo, the faces are now tagged in wrong positions when hovering over them in details panel.
It doesn't really break anything, but I find it a bit annoying.

[waclaw66]

Really missing such feature. Currently using SQL workaround...

delete from asset_faces where "assetId"='DESIRED_ASSET_ID'

Following Refresh thumbnails recognizes asset faces again. With caution!

[jhit]

I would love to see an option to share FR Data with a partner account. It is a huge effort to maintain and correct recognized faces. I want to share the training data and already tagged persons with my family. Furthermore, I want my family to be able to see and search by people on my shared pictures.

[aviv926]

#7045

[tuupola]

I am also looking for this feature. We have a group of people recognizing people from old photos. Currently this is possible only by using one shared account, which is not optimal.

[isaacolsen94]

Hey all, I've seen manually being able to tag faces mentioned a few times, but I don't see it on the list of features...is this something the team is looking into? Or is there no plan in implement it?

[alextran1502]

it is something we want to do eventually for sure

[isaacolsen94]

Awesome! That's good to hear! We know there is a lot of stuff on your plate, thanks for confirming!

[tuupola]

Manually tagging a person to a face would be great. For example if I have four photos of a person, three of them are recognized and person is created. Currently there is no easy way to add the fourth photo to the person.

Only way I can make this work is to set the MIN RECOGNIZED FACES faces to 1 and then keep merging the single image persons. Or maybe I am missing something?

[MacBadger]

Manual tagging is my number one! It would be great. The only feature Immich doesn't have!

But just to highlight at the top, "remove/update a wrongly tagged person" is ticked as complete. This isn't the case as far as i can tell. You can reassign. If say a none-person (or stranger!) is detected as being someone, you can't remove the tag... only change it to someone else.

[whoindahouse]

Any way to increase the number of unnamed faces you can merge into a named face from 5? Even temporarily...very useful when first scanning a library