Upload always stuck at ~340MB or 60s for large files on iOS

[Thinkscape]

The bug

Image upload seems to always get stuck at around 340MB or 60s.

The OS that Immich Server is running on

Fedora 39

Version of Immich Server

1.102.3

Version of Immich Mobile App

1.102.2 build.151

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

version: "3.2"

services:
  immich_server:
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start.sh", "immich"]
    volumes:
      - /var/data/immich/upload:/usr/src/app/upload
      - /dev/dri:/dev/dri
      - /etc/localtime:/etc/localtime:ro
    env_file: /var/data/config/immich/immich.env
    depends_on:
      - redis
      - database
    networks:
      - internal
      - traefik_public
    deploy:
      replicas: 1
      labels:
        # traefik
        - traefik.enable=true
        - traefik.docker.network=traefik_public

        # traefikv1
        - traefik.frontend.rule=Host:immich.example.com
        - traefik.port=3001

        # traefikv2
        - "traefik.http.routers.immich.rule=Host(`immich.example.com`)"
        - "traefik.http.routers.immich.entrypoints=https"
        - "traefik.http.services.immich.loadbalancer.server.port=3001"

  immich-microservices:
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start.sh", "microservices"]
    volumes:
      - /var/data/immich/upload:/usr/src/app/upload
      - /dev/dri:/dev/dri
      - /etc/localtime:/etc/localtime:ro
    env_file: /var/data/config/immich/immich.env
    depends_on:
      - redis
      - database
    networks:
      - internal
    deploy:
      labels:
        - traefik.enable=false

  immich-machine-learning:
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - /var/data/immich/cache:/cache
      - /dev/dri:/dev/dri
    env_file: /var/data/config/immich/immich.env
    networks:
      - internal
    deploy:
      labels:
        - traefik.enable=false

  redis:
    image: redis:6.2
    networks:
      - internal
    deploy:
      labels:
        - traefik.enable=false

  db:
    # image: postgres:14
    image: tensorchord/pgvecto-rs:pg14-v0.2.0
    env_file: /var/data/config/immich/immich.env
    volumes:
      - /var/data/runtime/immich/database:/var/lib/postgresql/data
    networks:
      - internal
    deploy:
      labels:
        - traefik.enable=false


networks:
  traefik_public:
    external: true
  internal:
    driver: overlay
    ipam:
      config:
        - subnet: 172.16.8.0/24

Your .env content

IMMICH_VERSION=release
PUID=1000
PGID=1000
TZ=Australia/Sydney

MACHINE_LEARNING_GPU_ACCELERATION= #optional
MACHINE_LEARNING_WORKERS=1 #optional
MACHINE_LEARNING_WORKER_TIMEOUT=120 #optional

###################################################################################
# Database
###################################################################################

# These are for the Immich components
DB_HOSTNAME=db
DB_USERNAME=postgres
DB_PASSWORD=postgres
DB_DATABASE_NAME=immich

# These are specific to how the postgres image likes to receive its ENV vars
POSTGRES_PASSWORD=postgres
#POSTGRES_USER=postgres
POSTGRES_DB=immich

###################################################################################
# Redis
###################################################################################

REDIS_HOSTNAME=redis

# Optional Redis settings:
# REDIS_PORT=6379
# REDIS_DBINDEX=0
# REDIS_PASSWORD=
# REDIS_SOCKET=


###################################################################################
# JWT SECRET
###################################################################################

JWT_SECRET=randomstringthatissolongandpowerfulthatnoonecanguess

###################################################################################
# MAPBOX
####################################################################################

# ENABLE_MAPBOX is either true of false -> if true, you have to provide MAPBOX_KEY
ENABLE_MAPBOX=false
MAPBOX_KEY=

###################################################################################
# WEB - Required
###################################################################################

# This is the URL of your vm/server where you host Immich, so that the web frontend
# know where can it make the request to.
# For example: If your server IP address is 10.1.11.50, the environment variable will
# be IMMICH_API_URL_EXTERNAL=http://10.1.11.50:2283/api
# !CAUTION! THERE IS NO FORWARD SLASH AT THE END

IMMICH_API_URL_EXTERNAL=https://immich.example.com/api

NODE_ENV=production

Reproduction steps

1. Docker swarm, running latest immich
2. Traefik forward proxy, auto discovered, no buffering limits specified, standard config
3. Download the iOS app
4. Connect, auth
5. Create a test Photos album with a video file over 350MB
6. Upload in foreground

Relevant log output

1. Nothing on the main docker container, just bunch of websocket connect/disconnect
2. Nothing relevant on iOS - log level set to finest, it's all just "flushing 1 packets"...
3. Nothing on traefik.

Additional information

1. The connection is local (LAN), no CDN involved
2. Tried restarting and rebuilding everything
3. Tried force-closing and reinstalling the mobile app
4. The database is clean (new) and the installation is clean
5. ~7TB free space
6. When tried to upload the same file via web browser, I'm getting "Couldn't upload" "Api error"
   

[bo0tzz]

Please try with a direct connection as well, without a reverse proxy in the way.

[Thinkscape]

The size is coincidental, the time was not.

In my case Traefik was getting in the way and it was not easy debugging it or finding a workaround due to how its layered and number of timeouts its tracking per request.

Here's the solve:

[entryPoints.https]
  address = ":443"
  [entryPoints.https.transport]
    [entryPoints.https.transport.respondingTimeouts]
      readTimeout = 0
      writeTimeout = 0
      idleTimeout = 0

[Tavisco]

I was having the same exact problem and that indeed fixed it. Thank you so much for sharing the solution!