gphotos import + albums migration

[Snuupy]

gphotos import + albums

With this guide, you STILL have to add images that are in multiple albums manually (except for the first instance of that image, so hopefully this isn’t too many) can get albums imported now as of immich-app/CLI#49

prereqs:

• install mattwilson1024/google-photos-exif, exiftool, ffmpeg
• have a working immich server + API key

Instructions

1. download:

   1. all album folders (separately)
   2. all takeout files

2. extract:

   1. all album folders
   2. all takeout files

note: your files that are in albums will be duplicated, but immich will handle the deduplication and not upload it twice

3. create directory structure to:

   Takeout
   	Albums # unzipped albums, some are missing metadata
   	AlbumsError # album processing errors
   	AlbumsProcessed # processed albums in their respective folders
   	GooglePhotos # unprocessed google photos, all whether they are in albums or not
   	error # GooglePhotos that can't be processed
   	output # processed GooglePhotos
   	archive_browser.html
   

4. run mattwilson1024/google-photos-exif on EACH album to add missing EXIF data:

   yarn start \
   	--inputDir '/mnt/f/user google takeout/Takeout/Albums/ALBUMNAME' \
   	--outputDir '/mnt/f/user google takeout/Takeout/AlbumsProcessed/ALBUMNAME' \
   	--errorDir '/mnt/f/user google takeout/Takeout/AlbumsError/ALBUMNAME'
   

5. run mattwilson1024/google-photos-exif on GooglePhotos to add missing EXIF data:

   yarn start \
   	--inputDir '/mnt/f/user google takeout/Takeout/Google Photos' \
   	--outputDir '/mnt/f/user google takeout/Takeout/output' \
   	--errorDir '/mnt/f/user google takeout/Takeout/error'
   

6. if any have errored out in AlbumsProcessed or error, fix metadata by running:

   exiftool -all= -tagsfromfile @ -all:all -unsafe -icc_profile .

   then run mattwilson1024/google-photos-exifagain on the (fixed) error directory (2nd pass), they should all pass now. move all output files to their corresponding output folders

7. upload ALBUMS to immich FIRST so media gets put into albums (otherwise it will not be put inside their respective albums and you’ll have to manually select each photo manually to add to the album, and you can’t re-upload it because of deduplication which rejects files with the same hash) Edit: I prefer doing it this way, but this no longer /has/ to be done first as the CLI will still associate images with albums if they are in the folder structure.

   Upload each album INDIVIDUALLY and VERIFY file counts. If there are any discrepancies, figure out why. For example, if a file is not supported, some files are missing their corresponding json files so metadata was not added, or the file is a duplicate. If there were any errors, rerun mattwilson1024/google-photos-exif on them using the commands above.

   immich upload \
   	--key APIKEY \
   	--server https://immich.domain.tld/api \
   	-d '/mnt/f/user google takeout/Takeout/AlbumsProcessed/ALBUM' -t 16 -al # adds each folder into its own album

for unsupported files, you can convert the format using: ffmpeg -i example.MPG example.mp4 or ffmpeg -i filename.mov -c copy filename.mp4

for missing json files, you can find the nearest neighbor, such as longfilename.(1).jp note the missing “g” in jp (as opposed to jpg)

you can compare whether images have the same hash with:

sha1sum file.ext or find duplicates with https://github.com/qarmin/czkawka

for duplicate files in multiple albums, add the corresponding missing images back to their albums.

8. Once all albums have been processed and verified, upload the rest that are not in albums:

   immich upload \
   	--key APIKEY \
   	--server https://immich.domain.tld/api \
   	-d '/mnt/f/user google takeout/Takeout/output' -t 16

9. Verify you don’t have any files that show up as current day upon refreshing immich. Repeat any of the above steps if they are showing up without proper metadata.

Rejoice as you are now free from gphotos.

[Snuupy]

A comparison for takeout helper tools:

mattwilson1024/google-photos-exif:

• slow(er) but processed most files properly except the really really weird edgecases like: longfilename-hash.(1).jp.json (note the (1), jp instead of jpg, etc.)
• fixes exif data
• can handle most -edited photos

https://github.com/TheLastGimbus/GooglePhotosTakeoutHelper v3

• doesn't fix exif data, only last modified date

https://github.com/TheLastGimbus/GooglePhotosTakeoutHelper v2

• misses all the -edited files

https://github.com/m1rkwood/google-photos-takeout-scripts

• have to specify extensions manually, have to copy all the json files manually, feels too brittle/too many edgecases for me

https://github.com/laurentlbm/google-photos-takeout-date-fixer

• it promises to update exif data in-place but fails for files larger than ~500MB. Here is the error log:

snuupy@laptop:~/google-photos-takeout-date-fixer$ yarn start /mnt/f/user/Takeout
--- Finding supported media files (.jpeg, .jpg, .gif, .mp4, .mov, .heic) ---
Found 14517 .jpg files
Found 1015 .mp4 files
Found 19 .gif files
--- Processing media files ---
...
...
...
    Error: Temporary file already exists: /mnt/f/user/Takeout/Photos from 2014/20140602_210850.mp4_exiftool_tmp

Prior to running this script the file 20140602_210850.mp4_exiftool_tmp does not exist.

[simulot]

May I mention my own project:

https://github.com/simulot/immich-go

• really easy to run. One binary file to run, no dependencies
• upload images directly, very fast
• parse multi-zip takeout with no need to unzip them
• address few weird filename cases I'm aware
• stack
  • jpg and raw images
  • images of a burst

[lukehmcc]

This script is currently broken. Will un-strike-out when it's fixed.

And my own project:

https://github.com/lukehmcc/simple-gphotos-immich-migration

The script really is quite simple, just a combination of a lot of work that's been done here so far. It utilizes a forked version of MrYakobo/exif-wrapper which utilizes the new and improved https://github.com/garzj/google-photos-migrate over mattwilson1024/google-photos-exif.

[amokster88]

is your script working now mate?

[lukehmcc]

As the README says, the functionality has been moved to google-photos-migrate. This was completed a couple months ago. I still have plans to integrate this library into immich-cli, but have been time strapped. Happy migrating!

[Snuupy]

As I import my last remaining user, here are some notes:

https://github.com/starmessage/CrcCheckCopy is useful if your user isn't very organized and has some old copies lying around for some reason, and has duplicates, but one folder is newer than the other, or some files are corrupted because they 😱 keep buying SD cards to store photos instead of moving them over to a hard drive

https://github.com/mrishab/google-photos-delete-tool/blob/master/delete_photos.js is useful to delete all gphotos after you're done migrating

[starbuck93]

About step 5, running google-photos-exif on the top level ./Takeout/Google Photos directory...
My Takeout was organized a bit different, so I don't think I need to do that. I don't think I did anything differently after downloading my Takeout than just unzipping or untaring every bundle into the Takeout folder.
My Takeout folder:

Takeout
	GooglePhotos
                [Albums...long list of folders including:]
                Album Title #1
                [a lot more album titles]
		Photos from 2020 #several thousand files each with a .json metadata file
		Photos from 2021 #etc....
		AlbumsError
		AlbumsProcessed
		shared_album_comments.json
	archive_browser.html

So, all my "unorganized" photos just end up in those "Photos from [Year]" albums, I guess!

Update: For anyone reading this in the future, upload the "Photos from..." albums last. Any album that starts alphabetically later than "Photos..." will be messed up!

[starbuck93]

Yeah, that seems to fix it. This could be a bit of manual work... The date is encoded in the filename, so I will probably remove the prefix of these couple dozen 113 files, in this album alone, and tell exiftool that the correct date is in the filename. But in this example, here's what I did:

exiftool.exe "-CreateDate=2021:12:10 22:13:51-06:00" -api "QuickTimeUTC" PXL_20211210_001012327.NIGHT.mp4

based on this

[starbuck93]

Thanks @Snuupy! I'll check out that utility.
The json file is named the same, I really think it's just the "media created" field that's messing with it, since exiftool worked on the Date fields.. Here are the original files:

[Snuupy]

What's in the json file that it doesn't apply the date correctly? Strange.

[starbuck93]

It's pretty regular looking:

{
  "title": "PXL_20211210_001012327.NIGHT.mp4",
  "description": "",
  "imageViews": "18",
  "creationTime": {
    "timestamp": "1639095244",
    "formatted": "Dec 10, 2021, 12:14:04 AM UTC"
  },
  "photoTakenTime": {
    "timestamp": "1639095231",
    "formatted": "Dec 10, 2021, 12:13:51 AM UTC"
  },
  "geoData": {
    "redacted"
  },
  "geoDataExif": {
    "redacted"
  },
  "url": "redacted",
  "googlePhotosOrigin": {
    "mobileUpload": {
      "deviceFolder": {
        "localFolderName": ""
      },
      "deviceType": "ANDROID_PHONE"
    }
  },
  "photoLastModifiedTime": {
    "timestamp": "1660935461",
    "formatted": "Aug 19, 2022, 6:57:41 PM UTC"
  }
}

My best guess is the way that my Pixel 6 takes these pictures. This one is a "astrophotography" mode picture that is processed into a timelapse video shortly after it's finished processing the main photo. Based on the 113 other files in this album with the same issue, I believe that the Google Camera app is writing a bunch of zeros in the metadata field, or, the Takeout process adds them in there, but only to these processed videos.

[Snuupy]

That's certainly strange but I don't have the brainpower to work through why this is the case right now. Happy that you're able to set it manually instead in the meantime. Good thing we only have to do the migration once 😅

[uninvitedguest]

I'm trying to run this but running in to the same error regardless of whether I run yarn start or npm start. I don't understand why I'm running in to this, as run is definitely in the listed directory.

[Snuupy]

run yarn install first

[uninvitedguest]

Thanks very much for this, it got me through it all. After running the exif fixes and the import (used the docker run version of the import as I was performing it from the same device as the server) I found that ffmpeg was running full throttle long after I had finished - 80-90% core utilization - and I had no idea why. Is this this anything you encountered?

[Snuupy]

Go to Administration -> Jobs -> Video transcoding

If there are files still being processed due to the import process queue then of course it will still be using high CPU. Just because Immich CLI has completed does not mean the files are finished processing.

You can run docker compose logs -f to see what's happening in real-time.

[lukehmcc]

This is a project I have not yet seen: https://github.com/gilesknap/gphotos-sync
I will do some testing and see how easy it is to use and how well it works.

[Snuupy]

read: https://github.com/gilesknap/gphotos-sync#warning-google-api-issues

[lukehmcc]

Ah yeah that'll do it

[MrYakobo]

I wrote a python script that codifies steps 1-6, so you don't have to do it manually.
It also handles the folder structuring issue that @starbuck93 had. Read through the script, it ain't that long 👍

https://github.com/MrYakobo/exif-wrapper

After running it, inspect the error folders for debugging.

[MrYakobo]

It seems like you're using @lukehmcc fork; please follow the instructions from his readme... 👍

[jeverling]

There is an issue in the fork where replacing a submodule didn't end up in git completely.
Please give this fork a try, which also fixes passing of arguments to the script that migrates photos, it wasn't working for me otherwise.

https://github.com/jeverling/exif-wrapper

Just to see if that works better for you.

[MrMetacomCode]

@jeverling Tried it, got this error:

Attaching to fix_metadata
fix_metadata  | Traceback (most recent call last):
fix_metadata  |   File "/app/fix_metadata.py", line 125, in <module>
fix_metadata  |     main(args.rootdir)
fix_metadata  |   File "/app/fix_metadata.py", line 115, in main
fix_metadata  |     rootdir = glob(f"{rootdir}/Google*/")[0].rstrip('/')
fix_metadata  |               ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
fix_metadata  | IndexError: list index out of range
fix_metadata exited with code 1

[lukehmcc]

I'd appreciate a PR when any issues are found (:

[pratik0809]

Before trying to use @jeverling's fix, make sure to delete old docker containers + images: docker rm <container_id> and docker image rm exif-wrapper-metadata before trying to run docker-compose up. Otherwise you're just re-using the old docker image and will keep getting the error Couldn't find a package.json file in "/app".

Seems obvious but it took me a while before i remembered i needed to rebuild the image.

[brettvd]

It looks like this doesn't quite work specifically for a Pixel 7. My guess is it effects any other Pixels that do all of the image post-processing.

Looking into the code of the mattwilson1024/google-photos-exif tool, it populates both fileCreatedAt and fileModifiedAt with the value in photoTakenTime in the JSON. The problem with that in the Pixel 7 is there's actually a second value, creationTime, that correlates with the actual fileModifiedAt time for the local file (generally I've seen it's somewhere from 5-6 seconds later).

What this means is: after uploading all of the images to immich, when trying to sync with existing files in the mobile app it will see the two files as "different". When doing a backup, it doesn't upload those files (assumingly because they have the same name), but you will see a local and remote copy of every image taken by the Pixel 7.

Trying to work through how to resolve this currently

[brettvd]

Update: I was able to come up with a solution.

Basically, since server hashing is working properly, I ended up deleting every image taken since getting the Pixel 7 from the server (it was about 1200 images). After some uninstalling/reinstalling and clearing memory in the app, I was able to then back them all up from the phone directly. After that, everything was synced with no duplicates. After that, I used the CLI steps to only upload the albums, which I still had saved, and that linked the now-correctly-uploaded items to the proper albums.

[lukehmcc]

Want to add on to this. So after running MrYakobo's lovely python script. (I couldn't get the docker container working, but just using the script by itself as the Dockerfile describes works great). There's a good way to optimize uploading the albums.

BE CAREFUL HERE.

You have to check that the python script was successful before using a script to automate uploads to make sure that you don't lose files. An easy way to do this is by running the command find AlbumsError/ and make sure that no files are listed (only sub-directories). The python script automatically generates directories for all of the albums, but this isn't an issue unless these directories are populated with assets. If there is a failed conversion, refer to the steps in the original post.

So it is safe to move forward if the find command looks like this:

covalent@hetzner1:~/remote1TB/gphotos-migration/Takeout/Google Photos$ find AlbumsError/
AlbumsError/
AlbumsError/Untitled(3)
AlbumsError/Untitled(49)
AlbumsError/Untitled(180)
AlbumsError/Untitled(33)
AlbumsError/A named album
AlbumsError/Untitled(70)
AlbumsError/Untitled(24)
AlbumsError/Untitled(41)
AlbumsError/Untitled(93)
AlbumsError/Untitled(15)
AlbumsError/Untitled(84)
AlbumsError/Untitled(56)
AlbumsError/Untitled(78)
...

Now for the goods.

#!/bin/bash

KEY="YOURKEYHERE"
SERVER="https://photos.yourdomain.com/api"
BASE_DIR="/foo/bar/Takeout/Google Photos/AlbumsProcessed"

# Change the working directory to BASE_DIR
cd "$BASE_DIR"

for album in "$BASE_DIR"/*; do
  if [ -d "$album" ]; then
    echo "Uploading album: $album"
    immich upload \
      -y \
      --key "$KEY" \
      --server "$SERVER" \
      -d "$album" -t 16 -al
  fi
done

Voila! You can skip manually uploading every album!

[nep2ner]

Thanks this guide is awesome! By the way, here is a convenient way to do the albums manually when you have a lot of them.

for each album

• set the env variable (change album_name each time): export album="album_name"
• run this line (same line each time): yarn start --inputDir ~/Desktop/Takeout/Albums/"${album}" --outputDir ~/Desktop/Takeout/AlbumsProcessed/"${album}" --errorDir ~/Desktop/Takeout/AlbumsError/"${album}"

[nep2ner]

This process worked really well for me! Out of about 60,000 photos/videos, about 200 of them got upload as today's date and 15 of them (all being videos) got upload as Jan 1, 2036. That means over 59,500 photos/videos were successfully transferred from Google photos while retaining the correct dates.

The 200 listed as today's date doesn't bother me too much, but the 15 listed as Jan 1, 2036 are going to be really annoying because they'll stay up at the top of the photo feed until 2036. Does anyone know how I can manually fix their dates? Should I just download them, delete them from Immich, use some tool to modify their created date, then re-upload?

Btw I used the docker version of the Immich CLI when importing.

[bo0tzz]

Should I just download them, delete them from Immich, use some tool to modify their created date, then re-upload?

That is probably the best approach, yes. I recommend exiftool for that.

[nep2ner]

Wow thanks for that software recommendation. exiftool -AllDates='2023:05:18 23:00:00' -overwrite_original . worked like a charm!

[garzj]

google-photos-exif didn't quite do the job for me, so I wrote my own little tool google-photos-migrate, but use at your own risk

[AngeloChen14]

Tried your scripts and it works like a charm and also fixes many extension error of my pictures, thanks a lot for your work!

[lukehmcc]

lol

Done! Processed 4947 files.
Files migrated: 1651
Files failed: 3296```

[p0n1]

Maybe related to this issue garzj/google-photos-migrate#3

[GodTamIt]

For those who are lazy, I used some Python snippets that others can use in ipython or a Jupyter notebook. Run all of these with your CWD in the mattwilson1024/google-photos-exif repo.

A little setup:

import os
import subprocessing

Step 4

4. run `mattwilson1024/google-photos-exif` on EACH album to add missing EXIF data:
   ```shell
   yarn start \
   	--inputDir '/mnt/f/user google takeout/Takeout/Albums/ALBUMNAME' \
   	--outputDir '/mnt/f/user google takeout/Takeout/AlbumsProcessed/ALBUMNAME' \
   	--errorDir '/mnt/f/user google takeout/Takeout/AlbumsError/ALBUMNAME'
   ```

takeout = "/mnt/f/user google takeout/takeout"
albums = next(os.walk(os.path.join(takeout, "Albums/")))[1]

for album in albums:
    subprocess.run(["yarn", "start", "--inputDir", os.path.join(takeout, "Albums/", album), "--outputDir", os.path.join(takeout, "AlbumsProcessed/", album), "--errorDir", os.path.join(takeout, "AlbumsError/", album)])

Step 7

7. _**upload ALBUMS to immich FIRST**_ ~so media gets put into albums (otherwise it will not be put inside their respective albums and you’ll have to manually select each photo manually to add to the album, and you can’t re-upload it because of deduplication which rejects files with the same hash)~ Edit: I prefer doing it this way, but this no longer /has/ to be done first as the CLI will still associate images with albums if they are in the folder structure.
   Upload each album **INDIVIDUALLY** and **VERIFY** file counts. If there are any discrepancies, figure out why. For example, if a file is not supported, some files are missing their corresponding `json` files so metadata was not added, or the file is a duplicate. If there were any errors, rerun `mattwilson1024/google-photos-exif` on them using the commands above.
   ```shell
   immich upload \
   	--key APIKEY \
   	--server https://immich.domain.tld/api \
   	-d '/mnt/f/user google takeout/Takeout/AlbumsProcessed/ALBUM' -t 16 -al # adds each folder into its own album
   ```

takeout = "/mnt/f/user google takeout/takeout"
albums = next(os.walk(os.path.join(takeout, "AlbumsProcessed/")))[1]
key = "APIKEY"
server = "https://immich.domain.tld/api"

for album in albums:
    subprocess.run(["immich", "upload", "--yes", "--key", key, "--server", server, "-t", "16", "-al", "--recursive", os.path.join(takeout, "AlbumsProcessed/", album)])

[midsorbet]

Here's @lukehmcc's #1340 (comment) script for those using Docker that creates the albums and then uploads the photos

#!/bin/bash

KEY="YOURKEYHERE"
SERVER="https://photos.yourdomain.com/api"
BASE_DIR="/foo/bar/Takeout/Google Photos/AlbumsProcessed"

# Change the working directory to BASE_DIR
cd "$BASE_DIR"

for album in "$BASE_DIR"/*; do
if [ -d "$album" ]; then
  albumname=$(basename -- "$album")
  echo "Uploading album from $album as $albumname"
  sudo docker run -it --rm -v "$album:/import" ghcr.io/immich-app/immich-cli:latest upload --key "$KEY" --server "$SERVER" -t 16 -al "$albumname"
  fi
done

[rajeshhazari]

Thanks for the script , unfortunately there are some updates to immich-cli image and the above script does not work as expected for me with the below changed script:

#!/bin/bash

API_KEY="xxxxx"
SERVER="http://<server-name>:2283/api"
BASE_DIR="/mnt/hdd2_tshb_abd/gtakeout_folder/albums"

# Change the working directory to BASE_DIR
cd "$BASE_DIR"

for album in "$BASE_DIR"/*; do
echo $album
if [ -d "$album" ]; then
  albumname=$(basename -- "$album")
  echo "Uploading album from $album as $albumname"

   docker run -it --rm -v "$album:/import" ghcr.io/immich-app/immich-cli:latest login-key $SERVER $API_KEY  upload -a --recursive /import/  --album $albumname  
   
  fi
done

Does anyone know the correct upload command or please share the documentation for upload the photos using immich-cli.

Thanks.

[rajeshhazari]

Figured out the correct command from other post, Incase if anyone is looking for an update:
replace docker cmd with below

docker run -it --rm -v "/mnt/hdd2_tshb_abd/gtakeout_folder/albums/:/import" -e IMMICH_INSTANCE_URL=$SERVER -e IMMICH_API_KEY=$API_KEY -e IMMICH_RECURSIVE=true -e IMMICH_AUTO_CREATE_ALBUM=true ghcr.io/immich-app/immich-cli:latest upload -r /import

[lukehmcc]

This script is currently broken, working on fixing it.

To expand upon what @buntingun just contributed, here's a fun script that will take you from a directory on your server with takeout zip files to full migration in one line!

wget -Nnv https://raw.githubusercontent.com/lukehmcc/simple-gphotos-immich-migration/master/process_and_upload.sh && bash process_and_upload.sh; rm -f process_and_upload.sh

When this command is run it prompts the user for the API address, API key, base dir (where the takeout zips are), and docker dir (a directory that can safely run docker containers via compose, as you cannot do this on cifs mounts). Once those values are input it unzips, cleans metadata, and automatically uploads all albums and photos.

Here is the repo(lukehmcc/simple-gphotos-immich-migration).

The script really is quite simple, just a combination of a lot of work that's been done here so far. It utilizes a forked version of MrYakobo/exif-wrapper which utilizes the new and improved https://github.com/garzj/google-photos-migrate over mattwilson1024/google-photos-exif.

Lmk where you guys would like to bring this next. Could this be something that could be implemented into immich directly? If so how do you guys envision that?

[MrMetacomCode]

I got this error:

fix_metadata  | Processing album /mnt/Google Photos/Albums/Joe’s 25th Bday/...
fix_metadata  | yarn run v1.22.19
fix_metadata  | error Couldn't find a package.json file in "/app"
fix_metadata  | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
fix_metadata  | Traceback (most recent call last):
fix_metadata  |   File "/app/fix_metadata.py", line 125, in <module>
fix_metadata  |     main(args.rootdir)
fix_metadata  |   File "/app/fix_metadata.py", line 118, in main
fix_metadata  |     process_albums(rootdir)
fix_metadata  |   File "/app/fix_metadata.py", line 61, in process_albums
fix_metadata  |     run_garzj_google_photos_migrate(albumdir, outdir, errdir)
fix_metadata  |   File "/app/fix_metadata.py", line 25, in run_garzj_google_photos_migrate
fix_metadata  |     sh(
fix_metadata  |   File "/app/fix_metadata.py", line 14, in sh
fix_metadata  |     p.check_returncode()
fix_metadata  |   File "/usr/local/lib/python3.11/subprocess.py", line 502, in check_returncode
fix_metadata  |     raise CalledProcessError(self.returncode, self.args, self.stdout,
fix_metadata  | subprocess.CalledProcessError: Command '['yarn', 'start', '--inputDir', '/mnt/Google Photos/Albums/Joe’s 25th Bday/', '--outputDir', '/mnt/Google Photos/AlbumsProcessed/Joe’s 25th Bday', '--errorDir', '/mnt/Google Photos/AlbumsError/Joe’s 25th Bday']' returned non-zero exit status 1.
fix_metadata exited with code 1

It only did this for one album. It then went on to connect to my Immich server and supposedly upload everything, but when I looked there was no content uploaded (albums or photos):

Unable to find image 'ghcr.io/immich-app/immich-cli:latest' locally
latest: Pulling from immich-app/immich-cli
8663204ce13b: Pull complete
d19b13245430: Pull complete
e4a0f03ce8f3: Pull complete
e319ab3d00f8: Pull complete
6d3e861397ac: Pull complete
422d4ab7e400: Pull complete
77bbb31f2770: Pull complete
33d10bc9a095: Pull complete
bf878295e6a9: Pull complete
36dfe90391d0: Pull complete
Digest: sha256:long-string-idk-if-I'm-not-supposed-to-share
Status: Downloaded newer image for ghcr.io/immich-app/immich-cli:latest
Checking connectivity with Immich instance...
Server status: OK
Checking credentials...
Login status: OK
Successful authentication for user username@email.com
Indexing local assets...
No local assets found, exiting
Uploading album: Joe’s 25th Bday
Checking connectivity with Immich instance...
Server status: OK
Checking credentials...
Login status: OK
Successful authentication for user username@email.com
Indexing local assets...
No local assets found, exiting

[lukehmcc]

I found the issue and am working on it. Sorry for late reply.

[RyanzpLee]

When you download the albums, do you do this manually by going to google photos and clicking download all on a specific album, or do you do this via takeout?

[Snuupy]

The first. I wasn't sure if the takeout zips had the same album hierarchy but they might.

[MrMetacomCode]

I was able to upload all 75GB of my two separate Google Photos takeouts using this repo to process the metadata (ran once on each takeout) and this bash script to upload the AlbumsProcessed from both takeouts. Finally, I ran this command in PhotosProcessed for each takeout to upload the photos themselves:

cd /home/mrmetacom/Original\ Google\ Photos\ Takeouts/takeout-20230709T030133Z-001/Takeout/Google\ Photos/PhotosProcessed/
sudo docker run --network immich-app_default -it --rm -v "$(pwd):/import" ghcr.io/immich-app/immich-cli:latest upload --key 123456789 --server http://immich-server:3001

Edit: Looks like a bunch of pictures didn't have the correct metadata attached to them. About 900 assets have the wrong metadata which is about 0.6% of everything. Not bad, but I think I'm just going to try everything the manual way as these "easy" methods don't seem to be working.

[di5cord20]

I get a similar error and the script exits. All takeout zip files were unzipped and folders updated… that’s as far as it got.

@lukehmcc, any thoughts or suggestions to fix?
@jeverling , same issue happened when I tried your fork as well.

2023-07-30 03:49:06 Restructured 36 folders
2023-07-30 03:49:07 Restructured 48 folders
2023-07-30 03:49:07 Processing album /mnt/Google Photos/Albums/Lisa_s BD - 2012/...
2023-07-30 03:49:08 yarn run v1.22.19
2023-07-30 03:49:08 error Couldn't find a package.json file in "/app"
2023-07-30 03:49:08 info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
2023-07-30 03:49:08 Traceback (most recent call last):
2023-07-30 03:49:08   File "/app/fix_metadata.py", line 125, in 
2023-07-30 03:49:08     main(args.rootdir)
2023-07-30 03:49:08   File "/app/fix_metadata.py", line 118, in main
2023-07-30 03:49:08     process_albums(rootdir)
2023-07-30 03:49:08   File "/app/fix_metadata.py", line 61, in process_albums
2023-07-30 03:49:08     run_garzj_google_photos_migrate(albumdir, outdir, errdir)
2023-07-30 03:49:08   File "/app/fix_metadata.py", line 25, in run_garzj_google_photos_migrate
2023-07-30 03:49:08     sh(
2023-07-30 03:49:08   File "/app/fix_metadata.py", line 14, in sh
2023-07-30 03:49:08     p.check_returncode()
2023-07-30 03:49:08   File "/usr/local/lib/python3.11/subprocess.py", line 502, in check_returncode
2023-07-30 03:49:08     raise CalledProcessError(self.returncode, self.args, self.stdout,
2023-07-30 03:49:08 subprocess.CalledProcessError: Command '['yarn', 'start', '--inputDir', '/mnt/Google Photos/Albums/Lisa_s BD - 2012/', '--outputDir', '/mnt/Google Photos/AlbumsProcessed/Lisa_s BD - 2012', '--errorDir', '/mnt/Google Photos/AlbumsError/Lisa_s BD - 2012']' returned non-zero exit status 1. 

[di5cord20]

The script was humming along until this happened:

fix_metadata  | The output directory is not empty. Pass "-f" to force the operation.
fix_metadata  | The error directory is not empty. Pass "-f" to force the operation.
fix_metadata  | error Command failed with exit code 1.

I started it up again but it errored out with the same message on a different album.

Do I need to pass any parameters when running the sudo docker compose up ?

[Snuupy]

afaik no, but you should clear out your output and error dirs first

[di5cord20]

I do see the yarn.lock file

~/docker/exif-wrapper/google-photos-migrate$ ls
android-dups  LICENSE  package.json  README.md  src  tsconfig.json  yarn.lock

[Snuupy]

I do see the yarn.lock file

yes, after you re-cloned the repo

it was likely not there before

[Snuupy]

See, this is the problem with AIO (all-in-one) scripts, people don't know how to fix things when they hit an error because they need to do it the manual way first. The AIO scripts are generally not robust unless they've been tried/tested by many people (one is not sufficient).

If you do it the manual way first, you can ensure each step is done correctly before proceeding onto the next, fixing any errors and knowing where the error stemmed from instead of playing guess the error.

[di5cord20]

afaik no, but you should clear out your output and error dirs first

What do you mean? Should I be deleting subfolders of AlbumsProcessed and AlbumsError that are empty?

[Snuupy]

Should I be deleting subfolders of AlbumsProcessed and AlbumsError that are empty?

Yes. No files nor folders should exist under AlbumsProcessed (if that's what you're calling output) and the same for AlbumsError. That is why you're getting that error: The output directory is not empty etc.

[di5cord20]

Got it... and I'm starting to appreciate your advice of going back to your initial post and doing each step one at a time. Before I go that route, I'd like to try the AIO script one more time. So, even though some of the albums processed the first time I ran the script (I see just the image files in the AlbumsProcessed album folders), your saying I should just delete all of them and start again?

[Snuupy]

unless you're able to resume it from the point of failure instead of starting it fresh again

[MrMetacomCode]

I must be stupid. I'm trying to do step 4 and I just get this output:

Usage: yarn [options]

yarn: error: no such option: --inputDir

I'm running the command in the google-photos-exif cloned repo directory.

Solution: I had to install npm.
sudo apt install npm
sudo npm install --global yarn
sudo yarn install
Then you can run the yarn command successfully.

[Snuupy]

what is the command you're using? should be yarn start...

[MrMetacomCode]

yarn start --inputDir '/home/mrmetacom/Original-Google-Photos-Takeouts/takeout-20230709T030133Z-001/Takeout/Google-Photos/Airsoft' --outputDir '/home/mrmetacom/Original-Google-Photos-Takeouts/takeout-20230709T030133Z-001/Takeout/AlbumsProcessed/Airsoft' --errorDir '/home/mrmetacom/Original-Google-Photos-Takeouts/takeout-20230709T030133Z-001/Takeout/AlbumsError/Airsoft'

[Snuupy]

are you running the command from the working directory you cloned mattwilson1024/google-photos-exif into?

[MrMetacomCode]

Yes. I googled the error and am currently installing npm

[MrMetacomCode]

Figured it out and edited my reply with the solution.

[MrMetacomCode]

On step 4 I get this response even though there are json files that correspond to each media file and a metadata.json file. Any ideas why?

--- Finished processing media files: ---
0 files with extension .jpeg
0 files with extension .jpg
3 files with extension .heic
0 files with extension .gif
2 files with extension .mp4
0 files with extension .png
0 files with extension .avi
2 files with extension .mov
--- The file modified timestamp has been updated on all media files ---
--- We did not edit EXIF metadata for any of the files. This could be because all files already had a value set for the DateTimeOriginal field, or because we did not have a corresponding JSON file. ---

Edit: I uploaded the album to the server just to test if everything had the correct date and it did. Not sure why but I'll take it.

[Snuupy]

Any ideas why?

Yes, all your files already have existing EXIF data. There is no need for the tool to add them. If you have some files from other devices (say, older phones/cameras/devices) that did NOT have EXIF data, this will add that. For me I had ~300+ which would show up at the beginning of the Immich timeline so things would be out of order.

However, if you didn't run this tool, you would not know this for sure/guarantee the state of the files.

When this tool traverses a file that already has EXIF data, it will not touch it (this is a good thing). If it is missing, it will read from the json file to see what dates to add to it so it fixes its position in the timeline.

This tool does not correct for files that have the wrong EXIF date data. For example, I had 1 file that was malformed and had some date in the future (like the year 2060 or something). That one I had to correct myself manually and was immediately obvious because it showed up at the top of the immich timeline (lol)

I uploaded the album to the server just to test if everything had the correct date and it did.

Exactly. If you get We did not edit EXIF metadata for any of the files. This could be because all files already had a value set for the DateTimeOriginal field, or because we did not have a corresponding JSON file., then you are good to go! (even if that error message isn't as positive as you would like it to be)

Not sure why but I'll take it.

Explanation above should cover it. I'm glad you figured it out! :)

[MrMetacomCode]

Any idea why some of my Google Photos didn't come with json files and don't have a DateTimeOriginal field but show up with the correct data on Google Photos? Should I request another takeout?

[simulot]

Apparently your archive is big, and it should be split into several zip files, like for mine. I found some JSON in another zip than the jpg.

[MrMetacomCode]

Ah yup you're right. I'm just going to combine the two takeouts into one.

[simulot]

Hi, I have discovered this discussion a bit late...

Anyway, I wrote a tool to import google photos takeout files with albums when present, directly from the zipped files. It handles albums, handles duplicates, discard field videos, and many other options. No manual work is required. Easy installation

Details here: https://github.com/simulot/immich-go#readme
https://github.com/simulot/immich-go/releases

Reading this discussion, I have checked the timeline and the map. Sure enough, I have some pictured taken on 2036, and some others located in the gulf of Guinea.
I can investigate this, and see if can update immich assets with date taken and GPS found in JSON files.
Let me know

[leowinterde]

Thanks, your tool has saved me a lot of time.

[simulot]

You're welcome. I'm glad to read that!

[MrYakobo]

It seems like the problem space of taking care of Takeout.zip was bigger than anybody thought; that's why we now have a million different repos (including my shitty one) that promise to import your Google Photos into immich.

Maybe we should make a wiki page and populate it with all possible invariants.
Some people miss JSON files sometimes, have different directory names, etc. Possibly due to different takeout settings, user language, what camera was used, etc. Having all that information in one place would enable any brave soul to write a complete (tm) script for uploading to Immich correctly.

The situation right now is confusing for everybody I think... 👍

[bo0tzz]

Google takeouts are a mess, and as you correctly note there have been many attempts at dealing with it. I think the main challenge here is that a person only ever needs to deal with this once, and so nobody is motivated to maintain a piece of tooling for the longer term. If a way past that issue is found, I think dealing with the actual takeout files in an effective way won't be a big problem comparatively. It would be awesome if this could be done in a way that is not tied to Immich specifically, so that anybody can make use of it :D

[Snuupy]

feel free to add tools here #1340 (comment)

[simulot]

feel free to add tools here #1340 (comment)

done

[AngelaDMerkel]

This doesn't appear to work on live video which I have taken. There are tools which are capable of merging the videos into the photos to create a live photo, but this seems an unnecessarily complex solution.

[janstadt]

I've successfully ran the google-photos-migrate tool on 1.16tb of google takeout photos (took 10hrs on my underpowered NAS but made it through). Can anyone help me understand the folder structure after everything is complete? Both AlbumsProcessed and PhotosProcessed seem to be close to the same size (~500gb a piece). Are these after deduplication scan? Do both directories have the same photos in them but is the photos directory just a flat directory and the albums directory has the same content but in their original directories? Id love to see a results.json file that lists all the stats once completed that explains things a bit. I need to know what directory i need to upload to immich and i dont want to do another dedupe scan. Any help would be great.

[lukehmcc]

Everything in the AlbumsProcessed/ dir should be a duplicate of a photo in PhotosProcessed/. How it works is nearly every photo gets put in PhotosProcessed/ in order to generate the full timeline, then photos organized in AlbumsProcessed/ signify which album each photo should be in. While uploading, it will check the hashes of each image with what's already been imported so nothing duplicate will be imported.

[janstadt]

Thanks @lukehmcc. If i dont plan on using the cli to upload and plan on mounting an existing readonly library with these photos, would i be better off just mounting the AlbumsProcessed or should i stick with the PhotosProcessed?

[lukehmcc]

Ah okay. I'm honestly not sure how dedup works if it's externally mounted. I presume that it will still hash each image and only put it on the timeline once but I can't say for sure.

There likely are photos in PhotosProcessed/ that aren't in AlbumsProcessed/ unless you put everything in albums. (Well unless you migrated over the Photos from x directories, then it should be about the same with both).

[janstadt]

All my photos were in separate directories so i presume they would be located in both.

[lukehmcc]

I would try mounting both and seeing how it goes. I'd like to know.

[eil047]

I prefer to keep the original quality/format of my photos/videos and not degrade to the reduced Google Photos upload/download quality. So I would like to apply the Google Albums metadata to recover those albums in my Immich originals.
This seems like the better approach but I don't have the IT skills to do that :-/
Anyone else has tried this yet?

[simulot]

I'm not familiar with the takeout data yet, I'm a bit reluctant to having to download too many (compressed => unwanted) pictures. Ideally I would only want the (shared) 'albums metadata' = album_ids with their corresponding photos/filenames and apply this information to recreate my albums with my original uncompressed media in Immich

That's an use case I haven't envisaged. Should I add a feature that recreate albums from Google Photos archives or why not using the google apis to avoid the dully process of takeout data?

[eil047]

Yes that is conceptually speaking what I want: just extract the 'album metadata' from Google Photos (via apis?) and not having to extract the media themselves (because useless)

[simulot]

Added to the roadmap.
This is new topic for me, it may take some time to be completed

[evrial]

I don't understand why enable photo compression and make your life more complicated after? It's your own fault, deal with it

[Snuupy]

Hi @eil047, try using czkawka or czkawka docker and click similar images, it may find many duplicates and then you can just batch delete the lower quality copies before importing to immich.

[clementdelestre]

Hi,
I made a little batch script to import Google photos folders from Windows:
To run it, save the code in a .bat file. It needs some dependencies, such as yarn, immich, etc.

Be aware of the \ at the end of each path.

@echo off 

set KEY=MY_KEY
set SERVER=https://immich.myserver.com/api
set BASE_DIR=C:\Users\me\...\Takeout\
set INPUT_DIR=GooglePhotos\
set OUTPUT_DIR=output\
set ERROR_DIR=error\

echo %KEY%
echo %SERVER%
echo %BASE_DIR%

echo Starting copying files

FOR /d %%i IN (%BASE_DIR%%INPUT_DIR%*) DO (
    echo Processing folder %%~nxi
    yarn start --inputDir "%%i" --outputDir "%BASE_DIR%%OUTPUT_DIR%%%~nxi" --errorDir "%BASE_DIR%%ERROR_DIR%%%~nxi"
)

echo Starting uploading files

echo logout
cmd /c "immich logout"

echo login
cmd /c "immich login-key %SERVER% %KEY%"

FOR /d %%i IN (%BASE_DIR%%OUTPUT_DIR%*) DO (
    echo Processing folder %%~nxi
    cmd /c immich upload "%%i"
)

PAUSE

[Frankster-NL]

The entire discussion above focusses on date taken, but do any of the scripts also take care of migrating face data? Did quite some work on tagging all of my 150K photos with relevant gps and face recognition data and would not like to see it getting lost...

[MrSimmo]

Hopefully this is useful - I used the command line in the posts from this forum thread (exiftool) to fix mine:

https://exiftool.org/forum/index.php?topic=12602.0

and this

https://exiftool.org/forum/index.php?topic=11064.45

This updated the original files instead of creating copies.

[Frankster-NL]

Useful links, but all of them only focus on datetime and gps metadata. If I look in my JSON files I am especially interested in the "people" and "name" tags...

[MrSimmo]

Think you could probably get exiftool to read in those (if there is a metadata exif tag that matches those). Not sure if they exist though.

Don't want to divert the topic but this is what I found on the above: https://exiftool.org/TagNames/EXIF.html

[jwilleke]

There is a thread on this but I have not gotten it to work yet.
https://exiftool.org/forum/index.php?topic=12772.new