1 round trip for both uploading and determining asset duplication

[truongsinh]

Spin of from #8825

The current design, which separates checkBulkUpload and upload into two distinct operations, introduces unnecessary server roundtrips, resulting in both performance and functionality issues. A more streamlined approach would be to integrate these processes into a single endpoint. This could be efficiently managed using an x-immich-checksum header to initially assess the asset. If the checksum identifies an asset that already exists and belongs to the same user, the server can simply return an HTTP 304 status code, indicating no change and eliminating the need for an upload. If the checksum matches an asset that exists but belongs to a different user, the server can respond with an HTTP 201, indicating that the asset is created (but through server-side copying mechanisms like hard links or --reflink=always, instead of consuming body stream). This method reduces bandwidth and storage requirements by preventing unnecessary uploads, optimizing overall system performance and resource utilization.

[bo0tzz]

I like the idea. A potential problem that comes to mind is that the upload request body includes not just the asset data, but also other fields like favourite/archived flags and such. If we short-circuit because of the checksum header, there's no way to still read those but not the asset data right?

[truongsinh]

fields like favourite/archived flags and such

If this is the same owner, I think it would be the same way it is handled right now (because after checkBulkUpload determines an asset is duplicated, it's never going to be uploaded anyway)

Otherwise (different owner, but same asset/checksum), you are right. I see 2 paths for now:

• moving all other fields to HTTP header
• split the endpoint into 2, sequentially. Semantically, one is dedicated to uploading (and returning asset ID), and the other updates all the fields (based on returned asset ID)

I don't have a strong opinion, but am slightly leaning toward moving all other fields to the HTTP header, as I consider them "metadata" as well.

[truongsinh]

Actually, looking at the (web) code:

        deviceAssetId,
        deviceId: 'WEB',
        fileCreatedAt,
        fileModifiedAt: new Date(asset.lastModified).toISOString(),
        isFavorite: 'false',
        duration: '0:00:00.000000',

It seems only deviceAssetId and deviceId are relevant, all others can be infered/set on server. And I think deviceAssetId and deviceId are truly "metadata" that semantically makes more sense to be in HTTP header for an upload endpoint.

[bo0tzz]

I think we've deprecated the id fields and are planning to remove them, so let's not worry about those.

The timestamp fields are necessary because they hold the filesystem metadata timestamps, for files that do not have embedded (exif) timestamps.

I think these things are only a problem if we're linking files from another owner, if the asset is a duplicate inside the current user's library then indeed the semantics don't change from how things work right now. I think the hardlinking idea might be worth an entirely separate conversation.

[truongsinh]

I think the hardlinking idea might be worth an entirely separate conversation.

Yeah I'm open to that. Btw, there are 2 methods, hardlinking (applicable to all Linux FS) and copy reflink=always (applicable to CoW FS like BtrFS, ZFS and BcacheFS)