Machine learning service fails to start

[lhjt]

The bug

Trying to start the machine learning service results in the following error before termination:

 Error: Unable to configure handler 'console'

The OS that Immich Server is running on

Containerised on Linux Ubuntu 22.04.4 LTS

Version of Immich Server

v1.103.1

Version of Immich Mobile App

N/A

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    io.kompose.service: immich-machine-learning
  name: immich-machine-learning
spec:
  replicas: 1
  selector:
    matchLabels:
      io.kompose.service: immich-machine-learning
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        io.kompose.service: immich-machine-learning
    spec:
      containers:
        - image: ghcr.io/immich-app/immich-machine-learning:v1.103.1
          name: immich-machine-learning
          ports:
            - containerPort: 3003
          resources:
            limits:
              cpu: 1000m
              memory: 1Gi
            requests:
              cpu: 10m
              memory: 64Mi
          volumeMounts:
            - mountPath: /cache
              name: model-cache
      restartPolicy: Always
      volumes:
        - name: model-cache
          hostPath:
            path: /volumes/immich/model-cache

Your .env content

N/A

Reproduction steps

1. Apply the above kubernetes manifest
2. Observe that the container fails with the error

Relevant log output

Error: Unable to configure handler 'console'

Additional information

No response

[bo0tzz]

This seems like a problem with your container environment. You're using kubernetes? Can you provide more detail?

[lhjt]

Yep, running in kubernetes by just applying the same config as the docker-compose config but just translated into k8s manifests:

Running in k3s: version v1.27.4+k3s1 (36645e73)
Host OS: Ubuntu 22.04.4 LTS; 16t, 16GB RAM
Image: ghcr.io/immich-app/immich-machine-learning:v1.103.1

Are there any other details you would like me to provide?

[lhjt]

I've also tried to run with LOG_LEVEL set to debug and the output remains the same.

Edit: I've also gone ahead and tried to remove the --log-config-json log_conf.json arg from the start script, and I get the following error:

Error: class uri 'app.config.CustomUvicornWorker' invalid or not found:

[Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 111, in load_class
    mod = importlib.import_module('.'.join(components))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/src/app/config.py", line 77, in <module>
    settings = Settings()
               ^^^^^^^^^^
  File "pydantic/env_settings.py", line 40, in pydantic.env_settings.BaseSettings.__init__
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for Settings
port
  value is not a valid integer (type=type_error.integer)
]

[lhjt]

Apparently I had to have the env var MACHINE_LEARNING_PORT set on the machine learning service; setting it fixed the problem 🤔

[bo0tzz]

It should not be required to set that, so I'm not sure why you needed to 🤔

Anyways, rather than translating the compose file manually, can I suggest using the helm chart? https://immich.app/docs/install/kubernetes

[lhjt]

Running the helm chart results in these errors for microservices:

[lhjt]

Furthermore it doesn't look like I can set resource limits using the helm chart; is this meant to be the case?

[bo0tzz]

What values did you use for the chart?

it doesn't look like I can set resource limits

Sure you can: immich-app/immich-charts#30 (comment)