How I Migrated My Entire Photo Library to Immich Without Duplicates - Over 120.000 Photos

[BoBBer446]

In the era of digital memories, managing a massive photo library can become a challenging endeavor, especially when moving away from popular services like Google Photos. Recently, I embarked on a significant project: transferring my entire collection of over 120,000 photos, which I had amassed over the years on Google Photos, to Immich, a self-hosted alternative. The goal was not just to move these images but to do so efficiently, ensuring no duplicates cluttered the new environment. Here’s how I tackled this hefty task.

The Challenge

Immich offers a robust API for uploads, but relying solely on this could potentially lead to duplicate entries, a common issue when dealing with extensive photo collections stored across different devices and services. Additionally, external libraries available for handling file transfers typically don't perform duplicate checking, which was crucial for my project.

Moreover, the complexity was further increased as both my girlfriend and I use Immich. This meant that two user accounts had to be managed simultaneously, each with their unique datasets.

Developing a Solution

To address these challenges, I developed a Python script tailored to handle the intricacies of our specific needs. The script can be found on my GitHub repository: python_copy2_image_duplicate_copy.

Here’s a breakdown of how the script functions:

• Recursive Directory Traversal: The script starts by recursively reading all the files from the source directory, which in this case, was the directory containing the exported Google Photos.
• Hash Generation: For each file, the script generates a SHA-256 hash. This hash acts as a unique identifier for each photo, regardless of its filename or metadata, ensuring that content duplication can be accurately detected.
• Duplicate Checking: Before moving any file to the target directory, the script compares the generated hash against those of the files already present in the target directory. This pre-check prevents any duplicate files from being copied or moved.
• Efficient File Handling: Depending on whether files need to be copied or moved, the script performs the necessary operations, ensuring that the target directory is organized and free from duplicates.

Integration with Immich

To seamlessly integrate this solution into Immich, which runs in a Docker environment using docker-compose, I configured two new mounts according to the Immich documentation:

• One for my account: ${EXTERNAL_PATH_USER1}:/usr/src/app/external/USER1
• One for my girlfriend’s account: ${EXTERNAL_PATH_USER2}:/usr/src/app/external/USER2

The corresponding environmental variables were set in the Docker ENV file as follows:

• EXTERNAL_PATH_USER1=/var/lib/docker/bindmounts/ext_photos/USER1
• EXTERNAL_PATH_USER2=/var/lib/docker/bindmounts/ext_photos/USER2

These settings allowed each user to have a dedicated external library within Immich, directly linked to the directories populated by the script.

The Outcome

The script successfully populated our Immich libraries without any duplicates, making our transition from Google Photos smooth and efficient. Each user can now access their unique set of images through Immich’s user-friendly interface, enjoying quick and easy retrieval of their digital memories.

References:

https://immich.app/docs/features/libraries/
https://immich.app/docs/features/command-line-interface
#8853 (comment)

[aviv926]

interesting.
I wonder if something like https://github.com/qarmin/czkawka
Could it also do the job the way you wanted it to?