Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: changing DefaultFunctionReadinessTimeoutSeconds value and rebuild nuclio has no effect with platform local #2951

Open
2 tasks done
eatgreen opened this issue Jun 9, 2023 · 0 comments

Comments

@eatgreen
Copy link

eatgreen commented Jun 9, 2023

Nuclio Version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of Nuclio.

Issue Description

I have used nuclio with cvat for auto annotation. My use case needs a long run time for larger image passing through the inference model (the deployed nuclio function). The small test runs fine, but larger test got following errors:
HTTPConnectionPool(host='host.docker.internal', port=32768): Read timed out. (read timeout=120).

As suggested by issue: #2864, there is no effect in passing parameter readinessTimeoutSeconds through function.yaml. So I have cloned tag 1.9.5 and have hard coded the
const DefaultFunctionReadinessTimeoutSeconds = 120 under pkg/platformconfig/types.go to 1200. Then I rebuild nuctl and images thanks to the steps in #927. The image build and function deploy are all fine.

Finally, at the end of the

nuctl deploy --project-name cvat --path "./serverless/pytorch/my-function/nuclio/" --platform local
log, I have

...platform":{"attributes":{"mountMode":"volume","restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"readinessTimeoutSeconds":1200,"securityContext":{},"eventTimeout":"2000s"}}}}

although the value of readinessTimeoutSeconds is 1200, my function still halts after 120s and I received the same error of HTTPConnectionPool as before.

Expected Behavior

My function should not halt given that readinessTimeoutSeconds

Deployment Method

Docker

Nuclio Version

1.9.5

Additional Information

My function.yaml

metadata:
  name: {redacted}
  namespace: cvat
  annotations:
    name: {redacted}
    type: detector
    framework: pytorch
    spec: |
      [
       {redacted}
      ]

spec:
  description: {redacted}
  runtime: 'python:3.9'
  handler: main:handler
  eventTimeout: 2000s
  # readinessTimeoutSeconds: 2000

  build:
    Image: {redacted}
    baseImage: ubuntu:22.04
    noCache: false

    onbuildImage: docker.io/my_own_build_handler-builder-python-onbuild:latest-amd64

    directives:
      preCopy:
        - kind: ENV
          value: DEBIAN_FRONTEND=noninteractive
        - kind: RUN
          value: apt-get update
        - kind: RUN
          value: apt-get install -y git python3 python3-pip python3-venv ffmpeg libsm6 libxext6 tesseract-ocr poppler-utils wget tesseract-ocr-grc
        - kind: WORKDIR
          value: /opt/nuclio
        - kind: RUN
          value: python3 -m pip install --upgrade build
        - kind: RUN
          value: git clone {redacted}
        - kind: WORKDIR
          value: /opt/nuclio/{redacted}
        - kind: RUN
          value: pip3 install -r requirements.txt
        - kind: RUN
          value: pip3 install {redacted}
        - kind: RUN
          value: ln -s /usr/bin/pip3 /usr/local/bin/pip && ln -s /usr/bin/python3 /usr/bin/python

  triggers:
    myHttpTrigger:
      maxWorkers: 1
      kind: 'http'
      workerAvailabilityTimeoutMilliseconds: 10000
      attributes:
        maxRequestBodySize: 33554432 # 32MB

  platform:
    attributes:
      restartPolicy:
        name: always
        maximumRetryCount: 3
      mountMode: volume

Thanks!

@eatgreen eatgreen changed the title [Bug]: changing DefaultFunctionReadinessTimeoutSeconds value and rebuild nuctl has no effect with platform local [Bug]: changing DefaultFunctionReadinessTimeoutSeconds value and rebuild nuclio has no effect with platform local Jun 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant