image de-duplication by comparing faces images.

[Raghucharan16]

• face_recognition version:1.3.0
• Python version:3.11.6
• Operating System:Ubuntu(23.10)

Description

Requirement:
I have 2 folders containing n no.of images, and each image may have different faces in them. So I need to extract only unique faces from images folder.
I have a folder of images which have multiple faces in each image. So I need to extract every face from the image and if it is unique i need to save the unique face to unique_faces folder. Like wise i have to check every image in the folder and need to save unique faces in a unique_faces folder.

given below is nested loop which iterates every image from both folders {i extracted only faces from images and saved them in 2 folders} and uses face recognition to compare each image and on a folder of 100 images i'm getting 5-7 false positives and time is approximately 25 min. so I searched for other recognition libraries like insightface, deeepface, compreface like that. but they were quicker but false positives are more. I tried with threading too but some what increase in performance is there but not that much still taking 15min.
Primary Goal : To eliminate the duplicate the images.
are there anyone with a solution for this.

What I Did

 
 for img1_filename in os.listdir(folder_1):
    img1_path = os.path.join(folder_1, img1_filename)
    
    for img2_filename in os.listdir(folder_2):
        img2_path = os.path.join(folder_2, img2_filename)

        if verify_faces(img1_path, img2_path):
            # Save the matched image from folder_1 into the output folder
            output_img_path = os.path.join(output_folder, img2_filename)
            shutil.copyfile(img1_path, output_img_path)
            print(f"Match found: {img1_filename} and {img2_filename}. Saved as {output_img_path}")
            break  # If a match is found, break the inner loop
    else:
        print("No Match")