Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Cannot deploy serverless function in Azure VM #3228

Open
2 tasks done
jaroslavknotek opened this issue Apr 22, 2024 · 8 comments
Open
2 tasks done

[Bug]: Cannot deploy serverless function in Azure VM #3228

jaroslavknotek opened this issue Apr 22, 2024 · 8 comments

Comments

@jaroslavknotek
Copy link

jaroslavknotek commented Apr 22, 2024 •
edited

Nuclio Version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of Nuclio.

Issue Description

I followed cvat tutorial and deployed it sucessfully. To extend the capabilites of CVAT the tutorial says to use nuctl to build a function,
however I cannot deploy a serverless function from Azure VM (ubuntu 22.04 + GPU nvidia).

I am stuck running the script deploy_[gpu|cpu]link:

>> ./deploy_gpu.sh pytorch/facebookresearch/sam/
24.04.22 11:23:00.937 (I)                     nuctl Project created {"Name": "cvat", "Namespace": "nuclio"}
Deploying pytorch/facebookresearch/sam function...
24.04.22 11:23:01.412 (I)                     nuctl Deploying function {"name": "pth-facebookresearch-sam-vit-h"}
24.04.22 11:23:01.412 (I)                     nuctl Building {"builderKind": "docker", "versionInfo": "Label: 1.12.15, Git commit: f15adda17a1b87bb2ce23b726ec3b3eb75436c12, OS: linux, Arch: amd64, Go version: go1.21.7", "name": "pth-facebookresearch-sam-vit-h"}
24.04.22 11:23:01.687 (I)                     nuctl Staging files and preparing base images
24.04.22 11:23:01.687 (W)                     nuctl Python 3.8 runtime is deprecated and will soon not be supported. Migrate your code and use Python 3.9 runtime (`python:3.9`) or higher
24.04.22 11:23:01.687 (W)                     nuctl Using user provided base image, runtime interpreter version is provided by the base image {"baseImage": "ubuntu:22.04"}
24.04.22 11:23:01.687 (I)                     nuctl Building processor image {"registryURL": "", "taggedImageName": "cvat.pth.facebookresearch.sam.vit_h:latest-gpu"}
24.04.22 11:23:01.688 (I)     nuctl.platform.docker Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.12.15-amd64"}
24.04.22 11:23:09.706 (W)     nuctl.platform.docker Docker command outputted to stderr - this may result in errors {"workingDir": "/tmp/nuclio-build-1701614961/staging", "cmd": "docker build --network host --force-rm -t nuclio-onbuild-coj4h78orl741b8kek2g -f /tmp/nuclio-build-1701614961/staging/Dockerfile.onbuild   --build-arg NUCLIO_LABEL=1.12.15 --build-arg NUCLIO_ARCH=amd64 --build-arg NUCLIO_BUILD_LOCAL_HANDLER_DIR=handler  .", "stderr": "ERROR: could not find /tmp/nuclio-build-1701614961/staging: stat /tmp/nuclio-build-1701614961/staging: no such file or directory\n"}
24.04.22 11:23:09.707 (W)                     nuctl Failed to create a function; setting the function status {"err": "Failed to build processor image", "errVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/shellrunner.go:114\n\nCall stack:\nstdout:\n\nstderr:\nERROR: could not find /tmp/nuclio-build-1701614961/staging: stat /tmp/nuclio-build-1701614961/staging: no such file or directory\n\n    /nuclio/pkg/cmdrunner/shellrunner.go:114\nFailed to build\n    /nuclio/pkg/dockerclient/shell.go:119\nFailed to build onbuild image\n    .../pkg/containerimagebuilderpusher/docker.go:257\nFailed to copy objects from onbuild\n    .../pkg/containerimagebuilderpusher/docker.go:217\nFailed to build image artifacts\n    .../pkg/containerimagebuilderpusher/docker.go:66\nFailed to build processor image\n    /nuclio/pkg/processor/build/builder.go:267\nFailed to build processor image"}

Error - exit status 1
    /nuclio/pkg/cmdrunner/shellrunner.go:114

Call stack:
stdout:

stderr:
ERROR: could not find /tmp/nuclio-build-1701614961/staging: stat /tmp/nuclio-build-1701614961/staging: no such file or directory

    /nuclio/pkg/cmdrunner/shellrunner.go:114
Failed to build
    /nuclio/pkg/dockerclient/shell.go:119
Failed to build onbuild image
    .../pkg/containerimagebuilderpusher/docker.go:257
Failed to copy objects from onbuild
    .../pkg/containerimagebuilderpusher/docker.go:217
Failed to build image artifacts
    .../pkg/containerimagebuilderpusher/docker.go:66

For demonstration purposes, I typed the problematic line separately:

>> nuctl deploy --project-name cvat --path pytorch/facebookresearch/sam/nuclio/         --file pytorch/facebookresearch/sam/nuclio/function-gpu.yaml --platform local

[Same output as above]

Expected Behavior

The script will not complain about not being able to find stuff in /tmp

Deployment Method

Docker

Nuclio Version

1.11.24,1.12.15,1.13.0

Additional Information

Going through the other issues, I noticed that it may be a problem with either

  • tmp being at a different volume (which is not according to findmnt tool) or
  • tmp having too strict permission (which it doesn't - I ensured that by chmod 777 tmp)

Additionally, I tried this script on my home PC (not ubuntu but archlinux) where everything runs fine.

What is so different with VM that nuctl doesn't work as expected?
Is there anything I can do?
@rokatyy
Copy link
Contributor

rokatyy commented Apr 23, 2024

Hi @jaroslavknotek ,

Could you please run nuctl again with --verbose? Perhaps it will provide us with additional insights.

@jaroslavknotek
Copy link
Author

Hello @rokatyy,

thank you for response. I noticed that the verbose logs (see below) explicitely say that /tmp/nuclio-* folder are created sucessfully. However, after the command ends, the folder is not there (not sure if it does not appear at all or is subsequently removed),

I also tried to check if temp is where it should be:

$ systemd-path temporary
/tmp

Verbose logs:

$  nuctl deploy --project-name cvat --path pytorch/facebookresearch/sam/nuclio/ --file pytorch/facebookresearch/sam/nuclio/function-gpu.yaml --platform local
24.04.24 07:42:24.786 (I)                     nuctl Project created {"Name": "cvat", "Namespace": "nuclio"}
Deploying pytorch/facebookresearch/sam function...
24.04.24 07:42:24.934 (D)                     nuctl Initializing platform {"version": "Label: 1.13.0, Git commit: c4422eb772781fb50fbf017698aae96199d81388, OS: linux, Arch: amd64, Go version: go1.21.7", "platformName": "local"}
24.04.24 07:42:24.934 (D)     nuctl.platform.docker Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"/bin/cat /etc/nuclio/store/projects/nuclio/default.json\"","Stdout":"","Stderr":"","Env":null}}
24.04.24 07:42:25.043 (D)                     nuctl Created platform {"name": "local", "namespace": "nuclio"}
24.04.24 07:42:25.043 (D)                     nuctl Loading function config from file {"file": "pytorch/facebookresearch/sam//nuclio/function-gpu.yaml"}
24.04.24 07:42:25.046 (D)                     nuctl Successfully loaded function config {"functionConfig": {"metadata":{"name":"pth-facebookresearch-sam-vit-h","namespace":"cvat","annotations":{"animated_gif":"https://raw.githubusercontent.com/opencv/cvat/develop/site/content/en/images/hrnet_example.gif","framework":"pytorch","help_message":"The interactor allows to get a mask of an object using at least one positive, and any negative points inside it","min_neg_points":"0","min_pos_points":"1","name":"Segment Anything","spec":"","type":"interactor","version":"2"}},"spec":{"description":"Interactive object segmentation with Segment-Anything","handler":"main:handler","runtime":"python:3.8","env":[{"name":"PYTHONPATH","value":"/opt/nuclio/sam"}],"resources":{"limits":{"nvidia.com/gpu":"1"}},"triggers":{"myHttpTrigger":{"class":"","kind":"http","name":"","workerAvailabilityTimeoutMilliseconds":10000,"attributes":{"maxRequestBodySize":33554432},"maxWorkers":1}},"build":{"image":"cvat.pth.facebookresearch.sam.vit_h","baseImage":"ubuntu:22.04","directives":{"preCopy":[{"kind":"ENV","value":"NVIDIA_VISIBLE_DEVICES=all"},{"kind":"ENV","value":"NVIDIA_DRIVER_CAPABILITIES=compute,utility"},{"kind":"ENV","value":"DEBIAN_FRONTEND=noninteractive"},{"kind":"WORKDIR","value":"/opt/nuclio/sam"},{"kind":"RUN","value":"apt-get update && apt-get -y install curl git python3 python3-pip ffmpeg libsm6 libxext6"},{"kind":"RUN","value":"pip3 install torch torchvision torchaudio opencv-python pycocotools matplotlib onnxruntime onnx"},{"kind":"RUN","value":"pip3 install git+https://github.com/facebookresearch/segment-anything.git"},{"kind":"RUN","value":"curl -O https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth"},{"kind":"RUN","value":"ln -s /usr/bin/pip3 /usr/local/bin/pip && ln -s /usr/bin/python3 /usr/bin/python"}]}},"platform":{"attributes":{"mountMode":"volume","restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"eventTimeout":"30s"}}}
24.04.24 07:42:25.046 (D)                     nuctl Deploying function {"functionConfig": {"metadata":{"name":"pth-facebookresearch-sam-vit-h","namespace":"nuclio","labels":{"nuclio.io/project-name":"cvat"},"annotations":{"animated_gif":"https://raw.githubusercontent.com/opencv/cvat/develop/site/content/en/images/hrnet_example.gif","framework":"pytorch","help_message":"The interactor allows to get a mask of an object using at least one positive, and any negative points inside it","min_neg_points":"0","min_pos_points":"1","name":"Segment Anything","spec":"","type":"interactor","version":"2"}},"spec":{"description":"Interactive object segmentation with Segment-Anything","handler":"main:handler","runtime":"python:3.8","env":[{"name":"PYTHONPATH","value":"/opt/nuclio/sam"}],"resources":{"limits":{"nvidia.com/gpu":"1"}},"targetCPU":75,"triggers":{"myHttpTrigger":{"class":"","kind":"http","name":"","workerAvailabilityTimeoutMilliseconds":10000,"attributes":{"maxRequestBodySize":33554432},"maxWorkers":1}},"build":{"path":"pytorch/facebookresearch/sam//nuclio","functionConfigPath":"pytorch/facebookresearch/sam//nuclio/function-gpu.yaml","image":"cvat.pth.facebookresearch.sam.vit_h","baseImage":"ubuntu:22.04","directives":{"preCopy":[{"kind":"ENV","value":"NVIDIA_VISIBLE_DEVICES=all"},{"kind":"ENV","value":"NVIDIA_DRIVER_CAPABILITIES=compute,utility"},{"kind":"ENV","value":"DEBIAN_FRONTEND=noninteractive"},{"kind":"WORKDIR","value":"/opt/nuclio/sam"},{"kind":"RUN","value":"apt-get update && apt-get -y install curl git python3 python3-pip ffmpeg libsm6 libxext6"},{"kind":"RUN","value":"pip3 install torch torchvision torchaudio opencv-python pycocotools matplotlib onnxruntime onnx"},{"kind":"RUN","value":"pip3 install git+https://github.com/facebookresearch/segment-anything.git"},{"kind":"RUN","value":"curl -O https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth"},{"kind":"RUN","value":"ln -s /usr/bin/pip3 /usr/local/bin/pip && ln -s /usr/bin/python3 /usr/bin/python"}]}},"platform":{"attributes":{"mountMode":"volume","restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"readinessTimeoutSeconds":120,"eventTimeout":"30s"}}}
24.04.24 07:42:25.046 (W)            nuctl.platform MaxWorkers is deprecated and will be removed in v1.15.x, use NumWorkers instead
24.04.24 07:42:25.046 (D)            nuctl.platform Enriching disable default http trigger flag {"functionName": "pth-facebookresearch-sam-vit-h", "disableDefaultHttpTrigger": false}
24.04.24 07:42:25.046 (D)            nuctl.platform Populating resources with default values {"defaultContainerResources": {"requests":{},"limits":{}}}
24.04.24 07:42:25.046 (D)            nuctl.platform Populated resources with default values {"resources": "&ResourceRequirements{Limits:ResourceList{nvidia.com/gpu: {{1 0} {<nil>} 1 DecimalSI},},Requests:ResourceList{cpu: {{25 -3} {<nil>} 25m DecimalSI},memory: {{1048576 0} {<nil>} 1Mi BinarySI},},Claims:[]ResourceClaim{},}"}
24.04.24 07:42:25.046 (D)     nuctl.platform.docker Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"/bin/cat /etc/nuclio/store/projects/nuclio/cvat.json\"","Stdout":"","Stderr":"","Env":null}}
24.04.24 07:42:25.154 (D)     nuctl.platform.docker Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"/bin/cat /etc/nuclio/store/functions/nuclio/pth-facebookresearch-sam-vit-h.json\"","Stdout":"","Stderr":"","Env":null}}
24.04.24 07:42:25.254 (I)                     nuctl Deploying function {"name": "pth-facebookresearch-sam-vit-h"}
24.04.24 07:42:25.254 (I)                     nuctl Building {"builderKind": "docker", "versionInfo": "Label: 1.13.0, Git commit: c4422eb772781fb50fbf017698aae96199d81388, OS: linux, Arch: amd64, Go version: go1.21.7", "name": "pth-facebookresearch-sam-vit-h"}
24.04.24 07:42:25.255 (D)                     nuctl Function configuration found in directory {"configFilePath": "pytorch/facebookresearch/sam//nuclio/function-gpu.yaml"}
24.04.24 07:42:25.256 (D)                     nuctl Created base temporary dir {"dir": "/tmp/nuclio-build-1487453310"}
24.04.24 07:42:25.256 (D)                     nuctl Created temporary dir {"dir": "/tmp/nuclio-build-1487453310/staging"}
24.04.24 07:42:25.256 (D)                     nuctl Created staging dir {"dir": "/tmp/nuclio-build-1487453310/staging"}
24.04.24 07:42:25.323 (D)                     nuctl Enriched configuration {"functionConfig": {"metadata":{"name":"pth-facebookresearch-sam-vit-h","namespace":"nuclio","labels":{"nuclio.io/project-name":"cvat"},"annotations":{"animated_gif":"https://raw.githubusercontent.com/opencv/cvat/develop/site/content/en/images/hrnet_example.gif","framework":"pytorch","help_message":"The interactor allows to get a mask of an object using at least one positive, and any negative points inside it","min_neg_points":"0","min_pos_points":"1","name":"Segment Anything","spec":"","type":"interactor","version":"2"}},"spec":{"description":"Interactive object segmentation with Segment-Anything","handler":"main:handler","runtime":"python:3.8","env":[{"name":"PYTHONPATH","value":"/opt/nuclio/sam"}],"resources":{"limits":{"nvidia.com/gpu":"1"},"requests":{"cpu":"25m","memory":"1Mi"}},"targetCPU":75,"triggers":{"myHttpTrigger":{"class":"","kind":"http","name":"myHttpTrigger","numWorkers":1,"workerAvailabilityTimeoutMilliseconds":10000,"attributes":{"maxRequestBodySize":33554432},"maxWorkers":1}},"build":{"path":"/home/gpuuser/cvat/serverless/pytorch/facebookresearch/sam/nuclio","functionConfigPath":"pytorch/facebookresearch/sam//nuclio/function-gpu.yaml","image":"cvat.pth.facebookresearch.sam.vit_h","baseImage":"ubuntu:22.04","directives":{"preCopy":[{"kind":"ENV","value":"NVIDIA_VISIBLE_DEVICES=all"},{"kind":"ENV","value":"NVIDIA_DRIVER_CAPABILITIES=compute,utility"},{"kind":"ENV","value":"DEBIAN_FRONTEND=noninteractive"},{"kind":"WORKDIR","value":"/opt/nuclio/sam"},{"kind":"RUN","value":"apt-get update && apt-get -y install curl git python3 python3-pip ffmpeg libsm6 libxext6"},{"kind":"RUN","value":"pip3 install torch torchvision torchaudio opencv-python pycocotools matplotlib onnxruntime onnx"},{"kind":"RUN","value":"pip3 install git+https://github.com/facebookresearch/segment-anything.git"},{"kind":"RUN","value":"curl -O https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth"},{"kind":"RUN","value":"ln -s /usr/bin/pip3 /usr/local/bin/pip && ln -s /usr/bin/python3 /usr/bin/python"}]},"codeEntryType":"image"},"platform":{"attributes":{"mountMode":"volume","restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"readinessTimeoutSeconds":120,"securityContext":{},"disableDefaultHTTPTrigger":false,"eventTimeout":"30s"}}, "platform": "local", "dependantImagesRegistryURL": "", "outputImageFile": "", "processorImage": {}}
24.04.24 07:42:25.323 (D)                     nuctl Creating shadow function {"name": "pth-facebookresearch-sam-vit-h"}
24.04.24 07:42:25.323 (D)            nuctl.platform Populating resources with default values {"defaultContainerResources": {"requests":{},"limits":{}}}
24.04.24 07:42:25.323 (D)            nuctl.platform Populated resources with default values {"resources": "&ResourceRequirements{Limits:ResourceList{nvidia.com/gpu: {{1 0} {<nil>} 1 DecimalSI},},Requests:ResourceList{cpu: {{25 -3} {<nil>} 25m DecimalSI},memory: {{1048576 0} {<nil>} 1Mi BinarySI},},Claims:[]ResourceClaim{},}"}
24.04.24 07:42:25.323 (D)     nuctl.platform.docker Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"/bin/cat /etc/nuclio/store/projects/nuclio/cvat.json\"","Stdout":"","Stderr":"","Env":null}}
24.04.24 07:42:25.434 (D)               nuctl.store Writing file contents {"path": "/etc/nuclio/store/functions/nuclio/pth-facebookresearch-sam-vit-h.json", "contents": "{\"metadata\":{\"name\":\"pth-facebookresearch-sam-vit-h\",\"namespace\":\"nuclio\",\"labels\":{\"nuclio.io/project-name\":\"cvat\"},\"annotations\":{\"animated_gif\":\"https://raw.githubusercontent.com/opencv/cvat/develop/site/content/en/images/hrnet_example.gif\",\"framework\":\"pytorch\",\"help_message\":\"The interactor allows to get a mask of an object using at least one positive, and any negative points inside it\",\"min_neg_points\":\"0\",\"min_pos_points\":\"1\",\"name\":\"Segment Anything\",\"spec\":\"\",\"type\":\"interactor\",\"version\":\"2\"}},\"spec\":{\"description\":\"Interactive object segmentation with Segment-Anything\",\"handler\":\"main:handler\",\"runtime\":\"python:3.8\",\"env\":[{\"name\":\"PYTHONPATH\",\"value\":\"/opt/nuclio/sam\"}],\"resources\":{\"limits\":{\"nvidia.com/gpu\":\"1\"},\"requests\":{\"cpu\":\"25m\",\"memory\":\"1Mi\"}},\"targetCPU\":75,\"triggers\":{\"myHttpTrigger\":{\"class\":\"\",\"kind\":\"http\",\"name\":\"myHttpTrigger\",\"numWorkers\":1,\"workerAvailabilityTimeoutMilliseconds\":10000,\"attributes\":{\"maxRequestBodySize\":33554432},\"maxWorkers\":1}},\"build\":{\"functionConfigPath\":\"pytorch/facebookresearch/sam//nuclio/function-gpu.yaml\",\"image\":\"cvat.pth.facebookresearch.sam.vit_h\",\"baseImage\":\"ubuntu:22.04\",\"directives\":{\"preCopy\":[{\"kind\":\"ENV\",\"value\":\"NVIDIA_VISIBLE_DEVICES=all\"},{\"kind\":\"ENV\",\"value\":\"NVIDIA_DRIVER_CAPABILITIES=compute,utility\"},{\"kind\":\"ENV\",\"value\":\"DEBIAN_FRONTEND=noninteractive\"},{\"kind\":\"WORKDIR\",\"value\":\"/opt/nuclio/sam\"},{\"kind\":\"RUN\",\"value\":\"apt-get update \\u0026\\u0026 apt-get -y install curl git python3 python3-pip ffmpeg libsm6 libxext6\"},{\"kind\":\"RUN\",\"value\":\"pip3 install torch torchvision torchaudio opencv-python pycocotools matplotlib onnxruntime onnx\"},{\"kind\":\"RUN\",\"value\":\"pip3 install git+https://github.com/facebookresearch/segment-anything.git\"},{\"kind\":\"RUN\",\"value\":\"curl -O https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth\"},{\"kind\":\"RUN\",\"value\":\"ln -s /usr/bin/pip3 /usr/local/bin/pip \\u0026\\u0026 ln -s /usr/bin/python3 /usr/bin/python\"}]},\"codeEntryType\":\"image\"},\"platform\":{\"attributes\":{\"mountMode\":\"volume\",\"restartPolicy\":{\"maximumRetryCount\":3,\"name\":\"always\"}}},\"readinessTimeoutSeconds\":120,\"securityContext\":{},\"disableDefaultHTTPTrigger\":false,\"eventTimeout\":\"30s\"},\"status\":{\"state\":\"building\"}}"}
24.04.24 07:42:25.435 (D)     nuctl.platform.docker Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"mkdir -p /etc/nuclio/store/functions/nuclio && /bin/printenv NUCLIO_CONTENTS > /etc/nuclio/store/functions/nuclio/pth-facebookresearch-sam-vit-h.json\"","Stdout":"","Stderr":"","Env":{"NUCLIO_CONTENTS":"eyJtZXRhZGF0YSI6eyJuYW1lIjoicHRoLWZhY2Vib29rcmVzZWFyY2gtc2FtLXZpdC1oIiwibmFtZXNwYWNlIjoibnVjbGlvIiwibGFiZWxzIjp7Im51Y2xpby5pby9wcm9qZWN0LW5hbWUiOiJjdmF0In0sImFubm90YXRpb25zIjp7ImFuaW1hdGVkX2dpZiI6Imh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9vcGVuY3YvY3ZhdC9kZXZlbG9wL3NpdGUvY29udGVudC9lbi9pbWFnZXMvaHJuZXRfZXhhbXBsZS5naWYiLCJmcmFtZXdvcmsiOiJweXRvcmNoIiwiaGVscF9tZXNzYWdlIjoiVGhlIGludGVyYWN0b3IgYWxsb3dzIHRvIGdldCBhIG1hc2sgb2YgYW4gb2JqZWN0IHVzaW5nIGF0IGxlYXN0IG9uZSBwb3NpdGl2ZSwgYW5kIGFueSBuZWdhdGl2ZSBwb2ludHMgaW5zaWRlIGl0IiwibWluX25lZ19wb2ludHMiOiIwIiwibWluX3Bvc19wb2ludHMiOiIxIiwibmFtZSI6IlNlZ21lbnQgQW55dGhpbmciLCJzcGVjIjoiIiwidHlwZSI6ImludGVyYWN0b3IiLCJ2ZXJzaW9uIjoiMiJ9fSwic3BlYyI6eyJkZXNjcmlwdGlvbiI6IkludGVyYWN0aXZlIG9iamVjdCBzZWdtZW50YXRpb24gd2l0aCBTZWdtZW50LUFueXRoaW5nIiwiaGFuZGxlciI6Im1haW46aGFuZGxlciIsInJ1bnRpbWUiOiJweXRob246My44IiwiZW52IjpbeyJuYW1lIjoiUFlUSE9OUEFUSCIsInZhbHVlIjoiL29wdC9udWNsaW8vc2FtIn1dLCJyZXNvdXJjZXMiOnsibGltaXRzIjp7Im52aWRpYS5jb20vZ3B1IjoiMSJ9LCJyZXF1ZXN0cyI6eyJjcHUiOiIyNW0iLCJtZW1vcnkiOiIxTWkifX0sInRhcmdldENQVSI6NzUsInRyaWdnZXJzIjp7Im15SHR0cFRyaWdnZXIiOnsiY2xhc3MiOiIiLCJraW5kIjoiaHR0cCIsIm5hbWUiOiJteUh0dHBUcmlnZ2VyIiwibnVtV29ya2VycyI6MSwid29ya2VyQXZhaWxhYmlsaXR5VGltZW91dE1pbGxpc2Vjb25kcyI6MTAwMDAsImF0dHJpYnV0ZXMiOnsibWF4UmVxdWVzdEJvZHlTaXplIjozMzU1NDQzMn0sIm1heFdvcmtlcnMiOjF9fSwiYnVpbGQiOnsiZnVuY3Rpb25Db25maWdQYXRoIjoicHl0b3JjaC9mYWNlYm9va3Jlc2VhcmNoL3NhbS8vbnVjbGlvL2Z1bmN0aW9uLWdwdS55YW1sIiwiaW1hZ2UiOiJjdmF0LnB0aC5mYWNlYm9va3Jlc2VhcmNoLnNhbS52aXRfaCIsImJhc2VJbWFnZSI6InVidW50dToyMi4wNCIsImRpcmVjdGl2ZXMiOnsicHJlQ29weSI6W3sia2luZCI6IkVOViIsInZhbHVlIjoiTlZJRElBX1ZJU0lCTEVfREVWSUNFUz1hbGwifSx7ImtpbmQiOiJFTlYiLCJ2YWx1ZSI6Ik5WSURJQV9EUklWRVJfQ0FQQUJJTElUSUVTPWNvbXB1dGUsdXRpbGl0eSJ9LHsia2luZCI6IkVOViIsInZhbHVlIjoiREVCSUFOX0ZST05URU5EPW5vbmludGVyYWN0aXZlIn0seyJraW5kIjoiV09SS0RJUiIsInZhbHVlIjoiL29wdC9udWNsaW8vc2FtIn0seyJraW5kIjoiUlVOIiwidmFsdWUiOiJhcHQtZ2V0IHVwZGF0ZSBcdTAwMjZcdTAwMjYgYXB0LWdldCAteSBpbnN0YWxsIGN1cmwgZ2l0IHB5dGhvbjMgcHl0aG9uMy1waXAgZmZtcGVnIGxpYnNtNiBsaWJ4ZXh0NiJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoicGlwMyBpbnN0YWxsIHRvcmNoIHRvcmNodmlzaW9uIHRvcmNoYXVkaW8gb3BlbmN2LXB5dGhvbiBweWNvY290b29scyBtYXRwbG90bGliIG9ubnhydW50aW1lIG9ubngifSx7ImtpbmQiOiJSVU4iLCJ2YWx1ZSI6InBpcDMgaW5zdGFsbCBnaXQraHR0cHM6Ly9naXRodWIuY29tL2ZhY2Vib29rcmVzZWFyY2gvc2VnbWVudC1hbnl0aGluZy5naXQifSx7ImtpbmQiOiJSVU4iLCJ2YWx1ZSI6ImN1cmwgLU8gaHR0cHM6Ly9kbC5mYmFpcHVibGljZmlsZXMuY29tL3NlZ21lbnRfYW55dGhpbmcvc2FtX3ZpdF9oXzRiODkzOS5wdGgifSx7ImtpbmQiOiJSVU4iLCJ2YWx1ZSI6ImxuIC1zIC91c3IvYmluL3BpcDMgL3Vzci9sb2NhbC9iaW4vcGlwIFx1MDAyNlx1MDAyNiBsbiAtcyAvdXNyL2Jpbi9weXRob24zIC91c3IvYmluL3B5dGhvbiJ9XX0sImNvZGVFbnRyeVR5cGUiOiJpbWFnZSJ9LCJwbGF0Zm9ybSI6eyJhdHRyaWJ1dGVzIjp7Im1vdW50TW9kZSI6InZvbHVtZSIsInJlc3RhcnRQb2xpY3kiOnsibWF4aW11bVJldHJ5Q291bnQiOjMsIm5hbWUiOiJhbHdheXMifX19LCJyZWFkaW5lc3NUaW1lb3V0U2Vjb25kcyI6MTIwLCJzZWN1cml0eUNvbnRleHQiOnt9LCJkaXNhYmxlRGVmYXVsdEhUVFBUcmlnZ2VyIjpmYWxzZSwiZXZlbnRUaW1lb3V0IjoiMzBzIn0sInN0YXR1cyI6eyJzdGF0ZSI6ImJ1aWxkaW5nIn19"}}}
24.04.24 07:42:25.526 (I)                     nuctl Staging files and preparing base images
24.04.24 07:42:25.526 (D)                     nuctl Runtime provided handler objects to staging dir {"handlerDirObjectPaths": ["/home/gpuuser/cvat/serverless/pytorch/facebookresearch/sam/nuclio"], "handlerDirInStaging": "/tmp/nuclio-build-1487453310/staging/handler", "handlerDirIncludingSubpath": "/tmp/nuclio-build-1487453310/staging/handler"}
24.04.24 07:42:25.527 (W)                     nuctl Python 3.8 runtime is deprecated and will soon not be supported. Migrate your code and use Python 3.9 runtime (`python:3.9`) or higher
24.04.24 07:42:25.527 (W)                     nuctl Using user provided base image, runtime interpreter version is provided by the base image {"baseImage": "ubuntu:22.04"}
24.04.24 07:42:25.527 (D)                     nuctl Created processor Dockerfile {"dockerfileInfo": "# Multistage builds\n\n\n\n# From the base image\nFROM ubuntu:22.04\n\n\n\n# Old(er) Docker support - must use all build args\nARG NUCLIO_LABEL\nARG NUCLIO_ARCH\nARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n\n\n# Run the pre-copy directives\n\nENV NVIDIA_VISIBLE_DEVICES=all\n\nENV NVIDIA_DRIVER_CAPABILITIES=compute,utility\n\nENV DEBIAN_FRONTEND=noninteractive\n\nWORKDIR /opt/nuclio/sam\n\nRUN apt-get update && apt-get -y install curl git python3 python3-pip ffmpeg libsm6 libxext6\n\nRUN pip3 install torch torchvision torchaudio opencv-python pycocotools matplotlib onnxruntime onnx\n\nRUN pip3 install git+https://github.com/facebookresearch/segment-anything.git\n\nRUN curl -O https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth\n\nRUN ln -s /usr/bin/pip3 /usr/local/bin/pip && ln -s /usr/bin/python3 /usr/bin/python\n\n\n\n# Copy required objects from the suppliers\n\nCOPY artifacts/processor /usr/local/bin/processor\n\nCOPY artifacts/py /opt/nuclio/\n\nCOPY artifacts/py3.8-whl /opt/nuclio/whl\n\nCOPY artifacts/uhttpc /usr/local/bin/uhttpc\n\n\n\nCOPY handler /opt/nuclio\n\n\n\n# Readiness probe\nHEALTHCHECK --interval=1s --timeout=3s CMD /usr/local/bin/uhttpc --url http://127.0.0.1:8082/ready || exit 1\n\n\n# Run the post-copy directives\n\nRUN python /opt/nuclio/whl/$(basename /opt/nuclio/whl/pip-*.whl)/pip install pip --no-index --find-links /opt/nuclio/whl && python -m pip install nuclio-sdk msgpack --no-index --find-links /opt/nuclio/whl\n\n\n# Run processor with configuration and platform configuration\nCMD [ \"processor\" ]\n", "baseImageRegistry": "", "onbuildImageRegistry": "quay.io"}
24.04.24 07:42:25.527 (I)                     nuctl Building processor image {"registryURL": "", "taggedImageName": "cvat.pth.facebookresearch.sam.vit_h:latest"}
24.04.24 07:42:25.527 (I)     nuctl.platform.docker Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.13.0-amd64"}
24.04.24 07:42:25.527 (D) tl.platform.docker.runner Executing {"command": "docker pull quay.io/nuclio/handler-builder-python-onbuild:1.13.0-amd64"}
24.04.24 07:42:30.832 (D) tl.platform.docker.runner Command executed successfully {"output": "1.13.0-amd64: Pulling from nuclio/handler-builder-python-onbuild\n0c3e43628b6b: Pulling fs layer\nbdb4f63315f1: Pulling fs layer\ndbc9de303df6: Pulling fs layer\n31726c119a38: Pulling fs layer\n8bacc431b4ee: Pulling fs layer\n8b11d973bb49: Pulling fs layer\n73a003026dd4: Pulling fs layer\n8bacc431b4ee: Waiting\n8b11d973bb49: Waiting\n31726c119a38: Waiting\n73a003026dd4: Waiting\n0c3e43628b6b: Download complete\n0c3e43628b6b: Pull complete\nbdb4f63315f1: Verifying Checksum\nbdb4f63315f1: Download complete\ndbc9de303df6: Verifying Checksum\ndbc9de303df6: Download complete\n31726c119a38: Verifying Checksum\n31726c119a38: Download complete\n8bacc431b4ee: Verifying Checksum\n8bacc431b4ee: Download complete\n8b11d973bb49: Verifying Checksum\n8b11d973bb49: Download complete\nbdb4f63315f1: Pull complete\ndbc9de303df6: Pull complete\n31726c119a38: Pull complete\n8bacc431b4ee: Pull complete\n8b11d973bb49: Pull complete\n73a003026dd4: Verifying Checksum\n73a003026dd4: Download complete\n73a003026dd4: Pull complete\nDigest: sha256:4d73b690f3be017926012a4b1d06242dd5b6274bc5ffbb847e9b881ac57983ed\nStatus: Downloaded newer image for quay.io/nuclio/handler-builder-python-onbuild:1.13.0-amd64\nquay.io/nuclio/handler-builder-python-onbuild:1.13.0-amd64\n", "stderr": "", "exitCode": 0}
24.04.24 07:42:30.833 (D)            nuctl.platform Generated onbuild Dockerfile {"contents": "FROM quay.io/nuclio/handler-builder-python-onbuild:1.13.0-amd64\nARG NUCLIO_LABEL\nARG NUCLIO_ARCH\n"}
24.04.24 07:42:30.833 (D)     nuctl.platform.docker Building image {"buildOptions": {"Image":"nuclio-onbuild-cokbfpgorl7ccl5ttneg","ContextDir":"/tmp/nuclio-build-1487453310/staging","DockerfilePath":"/tmp/nuclio-build-1487453310/staging/Dockerfile.onbuild","NoCache":false,"Pull":false,"BuildArgs":{"NUCLIO_ARCH":"amd64","NUCLIO_BUILD_LOCAL_HANDLER_DIR":"handler","NUCLIO_LABEL":"1.13.0"},"BuildFlags":null}}
24.04.24 07:42:30.833 (D) tl.platform.docker.runner Executing {"command": "docker build --network host --force-rm -t nuclio-onbuild-cokbfpgorl7ccl5ttneg -f /tmp/nuclio-build-1487453310/staging/Dockerfile.onbuild   --build-arg NUCLIO_LABEL=1.13.0 --build-arg NUCLIO_ARCH=amd64 --build-arg NUCLIO_BUILD_LOCAL_HANDLER_DIR=handler  ."}
24.04.24 07:42:31.023 (D) tl.platform.docker.runner Failed to execute command {"output": "", "stderr": "ERROR: could not find /tmp/nuclio-build-1487453310/staging: stat /tmp/nuclio-build-1487453310/staging: no such file or directory\n", "exitCode": 1, "err": "exit status 1"}
24.04.24 07:42:31.023 (W)     nuctl.platform.docker Docker command outputted to stderr - this may result in errors {"workingDir": "/tmp/nuclio-build-1487453310/staging", "cmd": "docker build --network host --force-rm -t nuclio-onbuild-cokbfpgorl7ccl5ttneg -f /tmp/nuclio-build-1487453310/staging/Dockerfile.onbuild   --build-arg NUCLIO_LABEL=1.13.0 --build-arg NUCLIO_ARCH=amd64 --build-arg NUCLIO_BUILD_LOCAL_HANDLER_DIR=handler  .", "stderr": "ERROR: could not find /tmp/nuclio-build-1487453310/staging: stat /tmp/nuclio-build-1487453310/staging: no such file or directory\n"}
24.04.24 07:42:31.024 (D)                     nuctl Successfully cleaned up temporary dir {"dir": "/tmp/nuclio-build-1487453310"}
24.04.24 07:42:31.024 (W)                     nuctl Failed to create a function; setting the function status {"err": "Failed to build processor image", "errVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/shellrunner.go:114\n\nCall stack:\nstdout:\n\nstderr:\nERROR: could not find /tmp/nuclio-build-1487453310/staging: stat /tmp/nuclio-build-1487453310/staging: no such file or directory\n\n    /nuclio/pkg/cmdrunner/shellrunner.go:114\nFailed to build\n    /nuclio/pkg/dockerclient/shell.go:119\nFailed to build onbuild image\n    .../pkg/containerimagebuilderpusher/docker.go:257\nFailed to copy objects from onbuild\n    .../pkg/containerimagebuilderpusher/docker.go:217\nFailed to build image artifacts\n    .../pkg/containerimagebuilderpusher/docker.go:66\nFailed to build processor image\n    /nuclio/pkg/processor/build/builder.go:267\nFailed to build processor image"}
24.04.24 07:42:31.024 (D)               nuctl.store Writing file contents {"path": "/etc/nuclio/store/functions/nuclio/pth-facebookresearch-sam-vit-h.json", "contents": "{\"metadata\":{\"name\":\"pth-facebookresearch-sam-vit-h\",\"namespace\":\"nuclio\",\"labels\":{\"nuclio.io/project-name\":\"cvat\"},\"annotations\":{\"animated_gif\":\"https://raw.githubusercontent.com/opencv/cvat/develop/site/content/en/images/hrnet_example.gif\",\"framework\":\"pytorch\",\"help_message\":\"The interactor allows to get a mask of an object using at least one positive, and any negative points inside it\",\"min_neg_points\":\"0\",\"min_pos_points\":\"1\",\"name\":\"Segment Anything\",\"spec\":\"\",\"type\":\"interactor\",\"version\":\"2\"}},\"spec\":{\"description\":\"Interactive object segmentation with Segment-Anything\",\"handler\":\"main:handler\",\"runtime\":\"python:3.8\",\"env\":[{\"name\":\"PYTHONPATH\",\"value\":\"/opt/nuclio/sam\"}],\"resources\":{\"limits\":{\"nvidia.com/gpu\":\"1\"},\"requests\":{\"cpu\":\"25m\",\"memory\":\"1Mi\"}},\"targetCPU\":75,\"triggers\":{\"myHttpTrigger\":{\"class\":\"\",\"kind\":\"http\",\"name\":\"myHttpTrigger\",\"numWorkers\":1,\"workerAvailabilityTimeoutMilliseconds\":10000,\"attributes\":{\"maxRequestBodySize\":33554432},\"maxWorkers\":1}},\"build\":{\"functionConfigPath\":\"pytorch/facebookresearch/sam//nuclio/function-gpu.yaml\",\"image\":\"cvat.pth.facebookresearch.sam.vit_h\",\"baseImage\":\"ubuntu:22.04\",\"directives\":{\"preCopy\":[{\"kind\":\"ENV\",\"value\":\"NVIDIA_VISIBLE_DEVICES=all\"},{\"kind\":\"ENV\",\"value\":\"NVIDIA_DRIVER_CAPABILITIES=compute,utility\"},{\"kind\":\"ENV\",\"value\":\"DEBIAN_FRONTEND=noninteractive\"},{\"kind\":\"WORKDIR\",\"value\":\"/opt/nuclio/sam\"},{\"kind\":\"RUN\",\"value\":\"apt-get update \\u0026\\u0026 apt-get -y install curl git python3 python3-pip ffmpeg libsm6 libxext6\"},{\"kind\":\"RUN\",\"value\":\"pip3 install torch torchvision torchaudio opencv-python pycocotools matplotlib onnxruntime onnx\"},{\"kind\":\"RUN\",\"value\":\"pip3 install git+https://github.com/facebookresearch/segment-anything.git\"},{\"kind\":\"RUN\",\"value\":\"curl -O https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth\"},{\"kind\":\"RUN\",\"value\":\"ln -s /usr/bin/pip3 /usr/local/bin/pip \\u0026\\u0026 ln -s /usr/bin/python3 /usr/bin/python\"}]},\"codeEntryType\":\"image\"},\"platform\":{\"attributes\":{\"mountMode\":\"volume\",\"restartPolicy\":{\"maximumRetryCount\":3,\"name\":\"always\"}}},\"readinessTimeoutSeconds\":120,\"securityContext\":{},\"disableDefaultHTTPTrigger\":false,\"eventTimeout\":\"30s\"},\"status\":{\"state\":\"error\",\"message\":\"\\nError - exit status 1\\n    /nuclio/pkg/cmdrunner/shellrunner.go:114\\n\\nCall stack:\\nstdout:\\n\\nstderr:\\nERROR: could not find /tmp/nuclio-build-1487453310/staging: stat /tmp/nuclio-build-1487453310/staging: no such file or directory\\n\\n    /nuclio/pkg/cmdrunner/shellrunner.go:114\\nFailed to build\\n    /nuclio/pkg/dockerclient/shell.go:119\\nFailed to build onbuild image\\n    .../pkg/containerimagebuilderpusher/docker.go:257\\nFailed to copy objects from onbuild\\n    .../pkg/containerimagebuilderpusher/docker.go:217\\nFailed to build image artifacts\\n    .../pkg/containerimagebuilderpusher/docker.go:66\\nFailed to build processor image\\n    /nuclio/pkg/processor/build/builder.go:267\\n\"}}"}
24.04.24 07:42:31.024 (D)     nuctl.platform.docker Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"mkdir -p /etc/nuclio/store/functions/nuclio && /bin/printenv NUCLIO_CONTENTS > /etc/nuclio/store/functions/nuclio/pth-facebookresearch-sam-vit-h.json\"","Stdout":"","Stderr":"","Env":{"NUCLIO_CONTENTS":"eyJtZXRhZGF0YSI6eyJuYW1lIjoicHRoLWZhY2Vib29rcmVzZWFyY2gtc2FtLXZpdC1oIiwibmFtZXNwYWNlIjoibnVjbGlvIiwibGFiZWxzIjp7Im51Y2xpby5pby9wcm9qZWN0LW5hbWUiOiJjdmF0In0sImFubm90YXRpb25zIjp7ImFuaW1hdGVkX2dpZiI6Imh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9vcGVuY3YvY3ZhdC9kZXZlbG9wL3NpdGUvY29udGVudC9lbi9pbWFnZXMvaHJuZXRfZXhhbXBsZS5naWYiLCJmcmFtZXdvcmsiOiJweXRvcmNoIiwiaGVscF9tZXNzYWdlIjoiVGhlIGludGVyYWN0b3IgYWxsb3dzIHRvIGdldCBhIG1hc2sgb2YgYW4gb2JqZWN0IHVzaW5nIGF0IGxlYXN0IG9uZSBwb3NpdGl2ZSwgYW5kIGFueSBuZWdhdGl2ZSBwb2ludHMgaW5zaWRlIGl0IiwibWluX25lZ19wb2ludHMiOiIwIiwibWluX3Bvc19wb2ludHMiOiIxIiwibmFtZSI6IlNlZ21lbnQgQW55dGhpbmciLCJzcGVjIjoiIiwidHlwZSI6ImludGVyYWN0b3IiLCJ2ZXJzaW9uIjoiMiJ9fSwic3BlYyI6eyJkZXNjcmlwdGlvbiI6IkludGVyYWN0aXZlIG9iamVjdCBzZWdtZW50YXRpb24gd2l0aCBTZWdtZW50LUFueXRoaW5nIiwiaGFuZGxlciI6Im1haW46aGFuZGxlciIsInJ1bnRpbWUiOiJweXRob246My44IiwiZW52IjpbeyJuYW1lIjoiUFlUSE9OUEFUSCIsInZhbHVlIjoiL29wdC9udWNsaW8vc2FtIn1dLCJyZXNvdXJjZXMiOnsibGltaXRzIjp7Im52aWRpYS5jb20vZ3B1IjoiMSJ9LCJyZXF1ZXN0cyI6eyJjcHUiOiIyNW0iLCJtZW1vcnkiOiIxTWkifX0sInRhcmdldENQVSI6NzUsInRyaWdnZXJzIjp7Im15SHR0cFRyaWdnZXIiOnsiY2xhc3MiOiIiLCJraW5kIjoiaHR0cCIsIm5hbWUiOiJteUh0dHBUcmlnZ2VyIiwibnVtV29ya2VycyI6MSwid29ya2VyQXZhaWxhYmlsaXR5VGltZW91dE1pbGxpc2Vjb25kcyI6MTAwMDAsImF0dHJpYnV0ZXMiOnsibWF4UmVxdWVzdEJvZHlTaXplIjozMzU1NDQzMn0sIm1heFdvcmtlcnMiOjF9fSwiYnVpbGQiOnsiZnVuY3Rpb25Db25maWdQYXRoIjoicHl0b3JjaC9mYWNlYm9va3Jlc2VhcmNoL3NhbS8vbnVjbGlvL2Z1bmN0aW9uLWdwdS55YW1sIiwiaW1hZ2UiOiJjdmF0LnB0aC5mYWNlYm9va3Jlc2VhcmNoLnNhbS52aXRfaCIsImJhc2VJbWFnZSI6InVidW50dToyMi4wNCIsImRpcmVjdGl2ZXMiOnsicHJlQ29weSI6W3sia2luZCI6IkVOViIsInZhbHVlIjoiTlZJRElBX1ZJU0lCTEVfREVWSUNFUz1hbGwifSx7ImtpbmQiOiJFTlYiLCJ2YWx1ZSI6Ik5WSURJQV9EUklWRVJfQ0FQQUJJTElUSUVTPWNvbXB1dGUsdXRpbGl0eSJ9LHsia2luZCI6IkVOViIsInZhbHVlIjoiREVCSUFOX0ZST05URU5EPW5vbmludGVyYWN0aXZlIn0seyJraW5kIjoiV09SS0RJUiIsInZhbHVlIjoiL29wdC9udWNsaW8vc2FtIn0seyJraW5kIjoiUlVOIiwidmFsdWUiOiJhcHQtZ2V0IHVwZGF0ZSBcdTAwMjZcdTAwMjYgYXB0LWdldCAteSBpbnN0YWxsIGN1cmwgZ2l0IHB5dGhvbjMgcHl0aG9uMy1waXAgZmZtcGVnIGxpYnNtNiBsaWJ4ZXh0NiJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoicGlwMyBpbnN0YWxsIHRvcmNoIHRvcmNodmlzaW9uIHRvcmNoYXVkaW8gb3BlbmN2LXB5dGhvbiBweWNvY290b29scyBtYXRwbG90bGliIG9ubnhydW50aW1lIG9ubngifSx7ImtpbmQiOiJSVU4iLCJ2YWx1ZSI6InBpcDMgaW5zdGFsbCBnaXQraHR0cHM6Ly9naXRodWIuY29tL2ZhY2Vib29rcmVzZWFyY2gvc2VnbWVudC1hbnl0aGluZy5naXQifSx7ImtpbmQiOiJSVU4iLCJ2YWx1ZSI6ImN1cmwgLU8gaHR0cHM6Ly9kbC5mYmFpcHVibGljZmlsZXMuY29tL3NlZ21lbnRfYW55dGhpbmcvc2FtX3ZpdF9oXzRiODkzOS5wdGgifSx7ImtpbmQiOiJSVU4iLCJ2YWx1ZSI6ImxuIC1zIC91c3IvYmluL3BpcDMgL3Vzci9sb2NhbC9iaW4vcGlwIFx1MDAyNlx1MDAyNiBsbiAtcyAvdXNyL2Jpbi9weXRob24zIC91c3IvYmluL3B5dGhvbiJ9XX0sImNvZGVFbnRyeVR5cGUiOiJpbWFnZSJ9LCJwbGF0Zm9ybSI6eyJhdHRyaWJ1dGVzIjp7Im1vdW50TW9kZSI6InZvbHVtZSIsInJlc3RhcnRQb2xpY3kiOnsibWF4aW11bVJldHJ5Q291bnQiOjMsIm5hbWUiOiJhbHdheXMifX19LCJyZWFkaW5lc3NUaW1lb3V0U2Vjb25kcyI6MTIwLCJzZWN1cml0eUNvbnRleHQiOnt9LCJkaXNhYmxlRGVmYXVsdEhUVFBUcmlnZ2VyIjpmYWxzZSwiZXZlbnRUaW1lb3V0IjoiMzBzIn0sInN0YXR1cyI6eyJzdGF0ZSI6ImVycm9yIiwibWVzc2FnZSI6IlxuRXJyb3IgLSBleGl0IHN0YXR1cyAxXG4gICAgL251Y2xpby9wa2cvY21kcnVubmVyL3NoZWxscnVubmVyLmdvOjExNFxuXG5DYWxsIHN0YWNrOlxuc3Rkb3V0OlxuXG5zdGRlcnI6XG5FUlJPUjogY291bGQgbm90IGZpbmQgL3RtcC9udWNsaW8tYnVpbGQtMTQ4NzQ1MzMxMC9zdGFnaW5nOiBzdGF0IC90bXAvbnVjbGlvLWJ1aWxkLTE0ODc0NTMzMTAvc3RhZ2luZzogbm8gc3VjaCBmaWxlIG9yIGRpcmVjdG9yeVxuXG4gICAgL251Y2xpby9wa2cvY21kcnVubmVyL3NoZWxscnVubmVyLmdvOjExNFxuRmFpbGVkIHRvIGJ1aWxkXG4gICAgL251Y2xpby9wa2cvZG9ja2VyY2xpZW50L3NoZWxsLmdvOjExOVxuRmFpbGVkIHRvIGJ1aWxkIG9uYnVpbGQgaW1hZ2VcbiAgICAuLi4vcGtnL2NvbnRhaW5lcmltYWdlYnVpbGRlcnB1c2hlci9kb2NrZXIuZ286MjU3XG5GYWlsZWQgdG8gY29weSBvYmplY3RzIGZyb20gb25idWlsZFxuICAgIC4uLi9wa2cvY29udGFpbmVyaW1hZ2VidWlsZGVycHVzaGVyL2RvY2tlci5nbzoyMTdcbkZhaWxlZCB0byBidWlsZCBpbWFnZSBhcnRpZmFjdHNcbiAgICAuLi4vcGtnL2NvbnRhaW5lcmltYWdlYnVpbGRlcnB1c2hlci9kb2NrZXIuZ286NjZcbkZhaWxlZCB0byBidWlsZCBwcm9jZXNzb3IgaW1hZ2VcbiAgICAvbnVjbGlvL3BrZy9wcm9jZXNzb3IvYnVpbGQvYnVpbGRlci5nbzoyNjdcbiJ9fQ=="}}}
24.04.24 07:42:31.151 (D)                     nuctl Saving deployment logs
24.04.24 07:42:31.151 (D)     nuctl.platform.docker Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"/bin/cat /etc/nuclio/store/functions/nuclio/pth-facebookresearch-sam-vit-h.json\"","Stdout":"","Stderr":"","Env":null}}
24.04.24 07:42:31.267 (D)               nuctl.store Writing file contents {"path": "/etc/nuclio/store/functions/nuclio/pth-facebookresearch-sam-vit-h.json", "contents": "{\"metadata\":{\"name\":\"pth-facebookresearch-sam-vit-h\",\"namespace\":\"nuclio\",\"labels\":{\"nuclio.io/project-name\":\"cvat\"},\"annotations\":{\"animated_gif\":\"https://raw.githubusercontent.com/opencv/cvat/develop/site/content/en/images/hrnet_example.gif\",\"framework\":\"pytorch\",\"help_message\":\"The interactor allows to get a mask of an object using at least one positive, and any negative points inside it\",\"min_neg_points\":\"0\",\"min_pos_points\":\"1\",\"name\":\"Segment Anything\",\"spec\":\"\",\"type\":\"interactor\",\"version\":\"2\"}},\"spec\":{\"description\":\"Interactive object segmentation with Segment-Anything\",\"handler\":\"main:handler\",\"runtime\":\"python:3.8\",\"env\":[{\"name\":\"PYTHONPATH\",\"value\":\"/opt/nuclio/sam\"}],\"resources\":{\"limits\":{\"nvidia.com/gpu\":\"1\"},\"requests\":{\"cpu\":\"25m\",\"memory\":\"1Mi\"}},\"targetCPU\":75,\"triggers\":{\"myHttpTrigger\":{\"class\":\"\",\"kind\":\"http\",\"name\":\"myHttpTrigger\",\"numWorkers\":1,\"workerAvailabilityTimeoutMilliseconds\":10000,\"attributes\":{\"maxRequestBodySize\":33554432},\"maxWorkers\":1}},\"build\":{\"functionConfigPath\":\"pytorch/facebookresearch/sam//nuclio/function-gpu.yaml\",\"image\":\"cvat.pth.facebookresearch.sam.vit_h\",\"baseImage\":\"ubuntu:22.04\",\"directives\":{\"preCopy\":[{\"kind\":\"ENV\",\"value\":\"NVIDIA_VISIBLE_DEVICES=all\"},{\"kind\":\"ENV\",\"value\":\"NVIDIA_DRIVER_CAPABILITIES=compute,utility\"},{\"kind\":\"ENV\",\"value\":\"DEBIAN_FRONTEND=noninteractive\"},{\"kind\":\"WORKDIR\",\"value\":\"/opt/nuclio/sam\"},{\"kind\":\"RUN\",\"value\":\"apt-get update \\u0026\\u0026 apt-get -y install curl git python3 python3-pip ffmpeg libsm6 libxext6\"},{\"kind\":\"RUN\",\"value\":\"pip3 install torch torchvision torchaudio opencv-python pycocotools matplotlib onnxruntime onnx\"},{\"kind\":\"RUN\",\"value\":\"pip3 install git+https://github.com/facebookresearch/segment-anything.git\"},{\"kind\":\"RUN\",\"value\":\"curl -O https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth\"},{\"kind\":\"RUN\",\"value\":\"ln -s /usr/bin/pip3 /usr/local/bin/pip \\u0026\\u0026 ln -s /usr/bin/python3 /usr/bin/python\"}]},\"codeEntryType\":\"image\"},\"platform\":{\"attributes\":{\"mountMode\":\"volume\",\"restartPolicy\":{\"maximumRetryCount\":3,\"name\":\"always\"}}},\"readinessTimeoutSeconds\":120,\"securityContext\":{},\"disableDefaultHTTPTrigger\":false,\"eventTimeout\":\"30s\"},\"status\":{\"state\":\"error\",\"message\":\"\\nError - exit status 1\\n    /nuclio/pkg/cmdrunner/shellrunner.go:114\\n\\nCall stack:\\nstdout:\\n\\nstderr:\\nERROR: could not find /tmp/nuclio-build-1487453310/staging: stat /tmp/nuclio-build-1487453310/staging: no such file or directory\\n\\n    /nuclio/pkg/cmdrunner/shellrunner.go:114\\nFailed to build\\n    /nuclio/pkg/dockerclient/shell.go:119\\nFailed to build onbuild image\\n    .../pkg/containerimagebuilderpusher/docker.go:257\\nFailed to copy objects from onbuild\\n    .../pkg/containerimagebuilderpusher/docker.go:217\\nFailed to build image artifacts\\n    .../pkg/containerimagebuilderpusher/docker.go:66\\nFailed to build processor image\\n    /nuclio/pkg/processor/build/builder.go:267\\n\",\"logs\":[{\"level\":\"info\",\"message\":\"Deploying function\",\"name\":\"pth-facebookresearch-sam-vit-h\",\"time\":1713944545254.9717},{\"builderKind\":\"docker\",\"level\":\"info\",\"message\":\"Building\",\"name\":\"pth-facebookresearch-sam-vit-h\",\"time\":1713944545255.002,\"versionInfo\":\"Label: 1.13.0, Git commit: c4422eb772781fb50fbf017698aae96199d81388, OS: linux, Arch: amd64, Go version: go1.21.7\"},{\"level\":\"info\",\"message\":\"Staging files and preparing base images\",\"name\":\"deployer\",\"time\":1713944545526.6897},{\"level\":\"warn\",\"message\":\"Python 3.8 runtime is deprecated and will soon not be supported. Migrate your code and use Python 3.9 runtime (`python:3.9`) or higher\",\"name\":\"deployer\",\"time\":1713944545527.0825},{\"baseImage\":\"ubuntu:22.04\",\"level\":\"warn\",\"message\":\"Using user provided base image, runtime interpreter version is provided by the base image\",\"name\":\"deployer\",\"time\":1713944545527.1033},{\"level\":\"info\",\"message\":\"Building processor image\",\"name\":\"deployer\",\"registryURL\":\"\",\"taggedImageName\":\"cvat.pth.facebookresearch.sam.vit_h:latest\",\"time\":1713944545527.3142},{\"err\":\"Failed to build processor image\",\"errVerbose\":\"\\nError - exit status 1\\n    /nuclio/pkg/cmdrunner/shellrunner.go:114\\n\\nCall stack:\\nstdout:\\n\\nstderr:\\nERROR: could not find /tmp/nuclio-build-1487453310/staging: stat /tmp/nuclio-build-1487453310/staging: no such file or directory\\n\\n    /nuclio/pkg/cmdrunner/shellrunner.go:114\\nFailed to build\\n    /nuclio/pkg/dockerclient/shell.go:119\\nFailed to build onbuild image\\n    .../pkg/containerimagebuilderpusher/docker.go:257\\nFailed to copy objects from onbuild\\n    .../pkg/containerimagebuilderpusher/docker.go:217\\nFailed to build image artifacts\\n    .../pkg/containerimagebuilderpusher/docker.go:66\\nFailed to build processor image\\n    /nuclio/pkg/processor/build/builder.go:267\\nFailed to build processor image\",\"level\":\"warn\",\"message\":\"Failed to create a function; setting the function status\",\"name\":\"deployer\",\"time\":1713944551024.5522}]}}"}
24.04.24 07:42:31.267 (D)     nuctl.platform.docker Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"mkdir -p /etc/nuclio/store/functions/nuclio && /bin/printenv NUCLIO_CONTENTS > /etc/nuclio/store/functions/nuclio/pth-facebookresearch-sam-vit-h.json\"","Stdout":"","Stderr":"","Env":{"NUCLIO_CONTENTS":"eyJtZXRhZGF0YSI6eyJuYW1lIjoicHRoLWZhY2Vib29rcmVzZWFyY2gtc2FtLXZpdC1oIiwibmFtZXNwYWNlIjoibnVjbGlvIiwibGFiZWxzIjp7Im51Y2xpby5pby9wcm9qZWN0LW5hbWUiOiJjdmF0In0sImFubm90YXRpb25zIjp7ImFuaW1hdGVkX2dpZiI6Imh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9vcGVuY3YvY3ZhdC9kZXZlbG9wL3NpdGUvY29udGVudC9lbi9pbWFnZXMvaHJuZXRfZXhhbXBsZS5naWYiLCJmcmFtZXdvcmsiOiJweXRvcmNoIiwiaGVscF9tZXNzYWdlIjoiVGhlIGludGVyYWN0b3IgYWxsb3dzIHRvIGdldCBhIG1hc2sgb2YgYW4gb2JqZWN0IHVzaW5nIGF0IGxlYXN0IG9uZSBwb3NpdGl2ZSwgYW5kIGFueSBuZWdhdGl2ZSBwb2ludHMgaW5zaWRlIGl0IiwibWluX25lZ19wb2ludHMiOiIwIiwibWluX3Bvc19wb2ludHMiOiIxIiwibmFtZSI6IlNlZ21lbnQgQW55dGhpbmciLCJzcGVjIjoiIiwidHlwZSI6ImludGVyYWN0b3IiLCJ2ZXJzaW9uIjoiMiJ9fSwic3BlYyI6eyJkZXNjcmlwdGlvbiI6IkludGVyYWN0aXZlIG9iamVjdCBzZWdtZW50YXRpb24gd2l0aCBTZWdtZW50LUFueXRoaW5nIiwiaGFuZGxlciI6Im1haW46aGFuZGxlciIsInJ1bnRpbWUiOiJweXRob246My44IiwiZW52IjpbeyJuYW1lIjoiUFlUSE9OUEFUSCIsInZhbHVlIjoiL29wdC9udWNsaW8vc2FtIn1dLCJyZXNvdXJjZXMiOnsibGltaXRzIjp7Im52aWRpYS5jb20vZ3B1IjoiMSJ9LCJyZXF1ZXN0cyI6eyJjcHUiOiIyNW0iLCJtZW1vcnkiOiIxTWkifX0sInRhcmdldENQVSI6NzUsInRyaWdnZXJzIjp7Im15SHR0cFRyaWdnZXIiOnsiY2xhc3MiOiIiLCJraW5kIjoiaHR0cCIsIm5hbWUiOiJteUh0dHBUcmlnZ2VyIiwibnVtV29ya2VycyI6MSwid29ya2VyQXZhaWxhYmlsaXR5VGltZW91dE1pbGxpc2Vjb25kcyI6MTAwMDAsImF0dHJpYnV0ZXMiOnsibWF4UmVxdWVzdEJvZHlTaXplIjozMzU1NDQzMn0sIm1heFdvcmtlcnMiOjF9fSwiYnVpbGQiOnsiZnVuY3Rpb25Db25maWdQYXRoIjoicHl0b3JjaC9mYWNlYm9va3Jlc2VhcmNoL3NhbS8vbnVjbGlvL2Z1bmN0aW9uLWdwdS55YW1sIiwiaW1hZ2UiOiJjdmF0LnB0aC5mYWNlYm9va3Jlc2VhcmNoLnNhbS52aXRfaCIsImJhc2VJbWFnZSI6InVidW50dToyMi4wNCIsImRpcmVjdGl2ZXMiOnsicHJlQ29weSI6W3sia2luZCI6IkVOViIsInZhbHVlIjoiTlZJRElBX1ZJU0lCTEVfREVWSUNFUz1hbGwifSx7ImtpbmQiOiJFTlYiLCJ2YWx1ZSI6Ik5WSURJQV9EUklWRVJfQ0FQQUJJTElUSUVTPWNvbXB1dGUsdXRpbGl0eSJ9LHsia2luZCI6IkVOViIsInZhbHVlIjoiREVCSUFOX0ZST05URU5EPW5vbmludGVyYWN0aXZlIn0seyJraW5kIjoiV09SS0RJUiIsInZhbHVlIjoiL29wdC9udWNsaW8vc2FtIn0seyJraW5kIjoiUlVOIiwidmFsdWUiOiJhcHQtZ2V0IHVwZGF0ZSBcdTAwMjZcdTAwMjYgYXB0LWdldCAteSBpbnN0YWxsIGN1cmwgZ2l0IHB5dGhvbjMgcHl0aG9uMy1waXAgZmZtcGVnIGxpYnNtNiBsaWJ4ZXh0NiJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoicGlwMyBpbnN0YWxsIHRvcmNoIHRvcmNodmlzaW9uIHRvcmNoYXVkaW8gb3BlbmN2LXB5dGhvbiBweWNvY290b29scyBtYXRwbG90bGliIG9ubnhydW50aW1lIG9ubngifSx7ImtpbmQiOiJSVU4iLCJ2YWx1ZSI6InBpcDMgaW5zdGFsbCBnaXQraHR0cHM6Ly9naXRodWIuY29tL2ZhY2Vib29rcmVzZWFyY2gvc2VnbWVudC1hbnl0aGluZy5naXQifSx7ImtpbmQiOiJSVU4iLCJ2YWx1ZSI6ImN1cmwgLU8gaHR0cHM6Ly9kbC5mYmFpcHVibGljZmlsZXMuY29tL3NlZ21lbnRfYW55dGhpbmcvc2FtX3ZpdF9oXzRiODkzOS5wdGgifSx7ImtpbmQiOiJSVU4iLCJ2YWx1ZSI6ImxuIC1zIC91c3IvYmluL3BpcDMgL3Vzci9sb2NhbC9iaW4vcGlwIFx1MDAyNlx1MDAyNiBsbiAtcyAvdXNyL2Jpbi9weXRob24zIC91c3IvYmluL3B5dGhvbiJ9XX0sImNvZGVFbnRyeVR5cGUiOiJpbWFnZSJ9LCJwbGF0Zm9ybSI6eyJhdHRyaWJ1dGVzIjp7Im1vdW50TW9kZSI6InZvbHVtZSIsInJlc3RhcnRQb2xpY3kiOnsibWF4aW11bVJldHJ5Q291bnQiOjMsIm5hbWUiOiJhbHdheXMifX19LCJyZWFkaW5lc3NUaW1lb3V0U2Vjb25kcyI6MTIwLCJzZWN1cml0eUNvbnRleHQiOnt9LCJkaXNhYmxlRGVmYXVsdEhUVFBUcmlnZ2VyIjpmYWxzZSwiZXZlbnRUaW1lb3V0IjoiMzBzIn0sInN0YXR1cyI6eyJzdGF0ZSI6ImVycm9yIiwibWVzc2FnZSI6IlxuRXJyb3IgLSBleGl0IHN0YXR1cyAxXG4gICAgL251Y2xpby9wa2cvY21kcnVubmVyL3NoZWxscnVubmVyLmdvOjExNFxuXG5DYWxsIHN0YWNrOlxuc3Rkb3V0OlxuXG5zdGRlcnI6XG5FUlJPUjogY291bGQgbm90IGZpbmQgL3RtcC9udWNsaW8tYnVpbGQtMTQ4NzQ1MzMxMC9zdGFnaW5nOiBzdGF0IC90bXAvbnVjbGlvLWJ1aWxkLTE0ODc0NTMzMTAvc3RhZ2luZzogbm8gc3VjaCBmaWxlIG9yIGRpcmVjdG9yeVxuXG4gICAgL251Y2xpby9wa2cvY21kcnVubmVyL3NoZWxscnVubmVyLmdvOjExNFxuRmFpbGVkIHRvIGJ1aWxkXG4gICAgL251Y2xpby9wa2cvZG9ja2VyY2xpZW50L3NoZWxsLmdvOjExOVxuRmFpbGVkIHRvIGJ1aWxkIG9uYnVpbGQgaW1hZ2VcbiAgICAuLi4vcGtnL2NvbnRhaW5lcmltYWdlYnVpbGRlcnB1c2hlci9kb2NrZXIuZ286MjU3XG5GYWlsZWQgdG8gY29weSBvYmplY3RzIGZyb20gb25idWlsZFxuICAgIC4uLi9wa2cvY29udGFpbmVyaW1hZ2VidWlsZGVycHVzaGVyL2RvY2tlci5nbzoyMTdcbkZhaWxlZCB0byBidWlsZCBpbWFnZSBhcnRpZmFjdHNcbiAgICAuLi4vcGtnL2NvbnRhaW5lcmltYWdlYnVpbGRlcnB1c2hlci9kb2NrZXIuZ286NjZcbkZhaWxlZCB0byBidWlsZCBwcm9jZXNzb3IgaW1hZ2VcbiAgICAvbnVjbGlvL3BrZy9wcm9jZXNzb3IvYnVpbGQvYnVpbGRlci5nbzoyNjdcbiIsImxvZ3MiOlt7ImxldmVsIjoiaW5mbyIsIm1lc3NhZ2UiOiJEZXBsb3lpbmcgZnVuY3Rpb24iLCJuYW1lIjoicHRoLWZhY2Vib29rcmVzZWFyY2gtc2FtLXZpdC1oIiwidGltZSI6MTcxMzk0NDU0NTI1NC45NzE3fSx7ImJ1aWxkZXJLaW5kIjoiZG9ja2VyIiwibGV2ZWwiOiJpbmZvIiwibWVzc2FnZSI6IkJ1aWxkaW5nIiwibmFtZSI6InB0aC1mYWNlYm9va3Jlc2VhcmNoLXNhbS12aXQtaCIsInRpbWUiOjE3MTM5NDQ1NDUyNTUuMDAyLCJ2ZXJzaW9uSW5mbyI6IkxhYmVsOiAxLjEzLjAsIEdpdCBjb21taXQ6IGM0NDIyZWI3NzI3ODFmYjUwZmJmMDE3Njk4YWFlOTYxOTlkODEzODgsIE9TOiBsaW51eCwgQXJjaDogYW1kNjQsIEdvIHZlcnNpb246IGdvMS4yMS43In0seyJsZXZlbCI6ImluZm8iLCJtZXNzYWdlIjoiU3RhZ2luZyBmaWxlcyBhbmQgcHJlcGFyaW5nIGJhc2UgaW1hZ2VzIiwibmFtZSI6ImRlcGxveWVyIiwidGltZSI6MTcxMzk0NDU0NTUyNi42ODk3fSx7ImxldmVsIjoid2FybiIsIm1lc3NhZ2UiOiJQeXRob24gMy44IHJ1bnRpbWUgaXMgZGVwcmVjYXRlZCBhbmQgd2lsbCBzb29uIG5vdCBiZSBzdXBwb3J0ZWQuIE1pZ3JhdGUgeW91ciBjb2RlIGFuZCB1c2UgUHl0aG9uIDMuOSBydW50aW1lIChgcHl0aG9uOjMuOWApIG9yIGhpZ2hlciIsIm5hbWUiOiJkZXBsb3llciIsInRpbWUiOjE3MTM5NDQ1NDU1MjcuMDgyNX0seyJiYXNlSW1hZ2UiOiJ1YnVudHU6MjIuMDQiLCJsZXZlbCI6Indhcm4iLCJtZXNzYWdlIjoiVXNpbmcgdXNlciBwcm92aWRlZCBiYXNlIGltYWdlLCBydW50aW1lIGludGVycHJldGVyIHZlcnNpb24gaXMgcHJvdmlkZWQgYnkgdGhlIGJhc2UgaW1hZ2UiLCJuYW1lIjoiZGVwbG95ZXIiLCJ0aW1lIjoxNzEzOTQ0NTQ1NTI3LjEwMzN9LHsibGV2ZWwiOiJpbmZvIiwibWVzc2FnZSI6IkJ1aWxkaW5nIHByb2Nlc3NvciBpbWFnZSIsIm5hbWUiOiJkZXBsb3llciIsInJlZ2lzdHJ5VVJMIjoiIiwidGFnZ2VkSW1hZ2VOYW1lIjoiY3ZhdC5wdGguZmFjZWJvb2tyZXNlYXJjaC5zYW0udml0X2g6bGF0ZXN0IiwidGltZSI6MTcxMzk0NDU0NTUyNy4zMTQyfSx7ImVyciI6IkZhaWxlZCB0byBidWlsZCBwcm9jZXNzb3IgaW1hZ2UiLCJlcnJWZXJib3NlIjoiXG5FcnJvciAtIGV4aXQgc3RhdHVzIDFcbiAgICAvbnVjbGlvL3BrZy9jbWRydW5uZXIvc2hlbGxydW5uZXIuZ286MTE0XG5cbkNhbGwgc3RhY2s6XG5zdGRvdXQ6XG5cbnN0ZGVycjpcbkVSUk9SOiBjb3VsZCBub3QgZmluZCAvdG1wL251Y2xpby1idWlsZC0xNDg3NDUzMzEwL3N0YWdpbmc6IHN0YXQgL3RtcC9udWNsaW8tYnVpbGQtMTQ4NzQ1MzMxMC9zdGFnaW5nOiBubyBzdWNoIGZpbGUgb3IgZGlyZWN0b3J5XG5cbiAgICAvbnVjbGlvL3BrZy9jbWRydW5uZXIvc2hlbGxydW5uZXIuZ286MTE0XG5GYWlsZWQgdG8gYnVpbGRcbiAgICAvbnVjbGlvL3BrZy9kb2NrZXJjbGllbnQvc2hlbGwuZ286MTE5XG5GYWlsZWQgdG8gYnVpbGQgb25idWlsZCBpbWFnZVxuICAgIC4uLi9wa2cvY29udGFpbmVyaW1hZ2VidWlsZGVycHVzaGVyL2RvY2tlci5nbzoyNTdcbkZhaWxlZCB0byBjb3B5IG9iamVjdHMgZnJvbSBvbmJ1aWxkXG4gICAgLi4uL3BrZy9jb250YWluZXJpbWFnZWJ1aWxkZXJwdXNoZXIvZG9ja2VyLmdvOjIxN1xuRmFpbGVkIHRvIGJ1aWxkIGltYWdlIGFydGlmYWN0c1xuICAgIC4uLi9wa2cvY29udGFpbmVyaW1hZ2VidWlsZGVycHVzaGVyL2RvY2tlci5nbzo2NlxuRmFpbGVkIHRvIGJ1aWxkIHByb2Nlc3NvciBpbWFnZVxuICAgIC9udWNsaW8vcGtnL3Byb2Nlc3Nvci9idWlsZC9idWlsZGVyLmdvOjI2N1xuRmFpbGVkIHRvIGJ1aWxkIHByb2Nlc3NvciBpbWFnZSIsImxldmVsIjoid2FybiIsIm1lc3NhZ2UiOiJGYWlsZWQgdG8gY3JlYXRlIGEgZnVuY3Rpb247IHNldHRpbmcgdGhlIGZ1bmN0aW9uIHN0YXR1cyIsIm5hbWUiOiJkZXBsb3llciIsInRpbWUiOjE3MTM5NDQ1NTEwMjQuNTUyMn1dfX0="}}}

Error - exit status 1
    /nuclio/pkg/cmdrunner/shellrunner.go:114

Call stack:
stdout:

stderr:
ERROR: could not find /tmp/nuclio-build-1487453310/staging: stat /tmp/nuclio-build-1487453310/staging: no such file or directory

    /nuclio/pkg/cmdrunner/shellrunner.go:114
Failed to build
    /nuclio/pkg/dockerclient/shell.go:119
Failed to build onbuild image
    .../pkg/containerimagebuilderpusher/docker.go:257
Failed to copy objects from onbuild
    .../pkg/containerimagebuilderpusher/docker.go:217
Failed to build image artifacts
    .../pkg/containerimagebuilderpusher/docker.go:66
``

@jaroslavknotek
Copy link
Author

Hello, is there any progress? I think I could be able to provide temporary access to the Azure VM so you can look around for yourself.

@jaroslavknotek
Copy link
Author

Hello. Is there any progress or reaction. My offer to help you debug still stands.

@rokatyy
Copy link
Contributor

rokatyy commented May 10, 2024

@jaroslavknotek Hey, sorry for the delay. I have a question—could you please verify during the nuctl execution if the file was actually created (just list files in directory with ls )? As far as I remember, Azure might perform cleanups in the temp directory occasionally, but I doubt it happens so frequently.

@jaroslavknotek
Copy link
Author

I read about handling TMP in Azure and it doesn't seem to be the culprit. Additionally, TMP has other stuff in there that hasn't been removed. TMP hadn't been empty before installation nor it was afterwards.

Considering watching the tmp content. I tried using ls and it didn't print out any nuclio-built-* folders. Also, I am not sure that ls is the correct command here becaus I would need to know the specific step when nuclio creates the folder and type ls before it gets supposedly deleted. Anyway neither ls nor watch ls did not help as it didn't prit nout any nuclio-built folder.

Is there anyhing else I can do to help you investigate?

@rokatyy
Copy link
Contributor

rokatyy commented May 13, 2024

@jaroslavknotek I found this issue on cvat gh - cvat-ai/cvat#2067 (comment). Could you please try cleaning up nuclio-local-storage-reader?

@jaroslavknotek
Copy link
Author

jaroslavknotek commented May 16, 2024 •
edited

@rokatyy Thank you for your response. Unfortunatelly, that didn't work. I also tried to deploy different functions to rule out that the problem is in SAM but to no avail.

To try your solution, I exec'd into the nuclio-local-storage-reader, found and deleted the artifact

$ docker exec ce88 ls /etc/nuclio/store/functions/nuclio/
pth-facebookresearch-sam-vit-h.json
$ docker exec ce88 rm /etc/nuclio/store/functions/nuclio/pth-facebookresearch-sam-vit-h.json
$ docker exec ce88 ls /etc/nuclio/store/functions/nuclio/
# nothing

Then I ran the command again and got the same error (see below)

I am surprised that you had to look into the github of CVAT. Can you tell me what is the problem here? I though that since I report error stemming from nuctl command then this is the correct repo to submit the issue to. I thought that the code responsible for deployment of nuclio service lies in this repository. The very same code that should be possible to alter so it logs what exactly it's doing so we will know where the problem is regardless whether it's CVAT or anything else what invokes it.

I understand that nuclio is open-source and I am grateful that you have been helping me. However, as a developer myself, I also think that a step leading to resolution of this issue would be to try nuclio on site (as I've offered you the machine) and see what the problem is and why it's not in the logs (My guess is that the confirmation that artifacts have been created would help here).

I am aware that I can do this myself and I considering getting into GO just to debug this problem but before that I'd like to hear from you. Do you think that we can resolve the problem with your development team or do I have to look into it. Also, I am not the only user stuck on this issue but I understand that the majority doesn't have any problem.

Thank you very much for your effort.

$ nuctl deploy --verbose --project-name cvat --path pytorch/facebookresearch/sam/nuclio/         --file pytorch/facebookresearch/sam/nuclio/function-gpu.yaml
24.05.16 07:41:55.952 (D)                     nuctl Initializing platform {"version": "Label: 1.13.0, Git commit: c4422eb772781fb50fbf017698aae96199d81388, OS: linux, Arch: amd64, Go version: go1.21.7", "platformName": "local"}
24.05.16 07:41:55.952 (D)     nuctl.platform.docker Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"/bin/cat /etc/nuclio/store/projects/nuclio/default.json\"","Stdout":"","Stderr":"","Env":null}}
24.05.16 07:41:56.071 (D)                     nuctl Created platform {"name": "local", "namespace": "nuclio"}
24.05.16 07:41:56.071 (D)                     nuctl Loading function config from file {"file": "pytorch/facebookresearch/sam/nuclio/function-gpu.yaml"}
24.05.16 07:41:56.073 (D)                     nuctl Successfully loaded function config {"functionConfig": {"metadata":{"name":"pth-facebookresearch-sam-vit-h","namespace":"cvat","annotations":{"animated_gif":"https://raw.githubusercontent.com/opencv/cvat/develop/site/content/en/images/hrnet_example.gif","framework":"pytorch","help_message":"The interactor allows to get a mask of an object using at least one positive, and any negative points inside it","min_neg_points":"0","min_pos_points":"1","name":"Segment Anything","spec":"","type":"interactor","version":"2"}},"spec":{"description":"Interactive object segmentation with Segment-Anything","handler":"main:handler","runtime":"python:3.8","env":[{"name":"PYTHONPATH","value":"/opt/nuclio/sam"}],"resources":{"limits":{"nvidia.com/gpu":"1"}},"triggers":{"myHttpTrigger":{"class":"","kind":"http","name":"","workerAvailabilityTimeoutMilliseconds":10000,"attributes":{"maxRequestBodySize":33554432},"maxWorkers":1}},"build":{"image":"cvat.pth.facebookresearch.sam.vit_h","baseImage":"ubuntu:22.04","directives":{"preCopy":[{"kind":"ENV","value":"NVIDIA_VISIBLE_DEVICES=all"},{"kind":"ENV","value":"NVIDIA_DRIVER_CAPABILITIES=compute,utility"},{"kind":"ENV","value":"DEBIAN_FRONTEND=noninteractive"},{"kind":"WORKDIR","value":"/opt/nuclio/sam"},{"kind":"RUN","value":"apt-get update && apt-get -y install curl git python3 python3-pip ffmpeg libsm6 libxext6"},{"kind":"RUN","value":"pip3 install torch torchvision torchaudio opencv-python pycocotools matplotlib onnxruntime onnx"},{"kind":"RUN","value":"pip3 install git+https://github.com/facebookresearch/segment-anything.git"},{"kind":"RUN","value":"curl -O https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth"},{"kind":"RUN","value":"ln -s /usr/bin/pip3 /usr/local/bin/pip && ln -s /usr/bin/python3 /usr/bin/python"}]}},"platform":{"attributes":{"mountMode":"volume","restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"eventTimeout":"30s"}}}
24.05.16 07:41:56.073 (D)                     nuctl Deploying function {"functionConfig": {"metadata":{"name":"pth-facebookresearch-sam-vit-h","namespace":"nuclio","labels":{"nuclio.io/project-name":"cvat"},"annotations":{"animated_gif":"https://raw.githubusercontent.com/opencv/cvat/develop/site/content/en/images/hrnet_example.gif","framework":"pytorch","help_message":"The interactor allows to get a mask of an object using at least one positive, and any negative points inside it","min_neg_points":"0","min_pos_points":"1","name":"Segment Anything","spec":"","type":"interactor","version":"2"}},"spec":{"description":"Interactive object segmentation with Segment-Anything","handler":"main:handler","runtime":"python:3.8","env":[{"name":"PYTHONPATH","value":"/opt/nuclio/sam"}],"resources":{"limits":{"nvidia.com/gpu":"1"}},"targetCPU":75,"triggers":{"myHttpTrigger":{"class":"","kind":"http","name":"","workerAvailabilityTimeoutMilliseconds":10000,"attributes":{"maxRequestBodySize":33554432},"maxWorkers":1}},"build":{"path":"pytorch/facebookresearch/sam/nuclio/","functionConfigPath":"pytorch/facebookresearch/sam/nuclio/function-gpu.yaml","image":"cvat.pth.facebookresearch.sam.vit_h","baseImage":"ubuntu:22.04","directives":{"preCopy":[{"kind":"ENV","value":"NVIDIA_VISIBLE_DEVICES=all"},{"kind":"ENV","value":"NVIDIA_DRIVER_CAPABILITIES=compute,utility"},{"kind":"ENV","value":"DEBIAN_FRONTEND=noninteractive"},{"kind":"WORKDIR","value":"/opt/nuclio/sam"},{"kind":"RUN","value":"apt-get update && apt-get -y install curl git python3 python3-pip ffmpeg libsm6 libxext6"},{"kind":"RUN","value":"pip3 install torch torchvision torchaudio opencv-python pycocotools matplotlib onnxruntime onnx"},{"kind":"RUN","value":"pip3 install git+https://github.com/facebookresearch/segment-anything.git"},{"kind":"RUN","value":"curl -O https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth"},{"kind":"RUN","value":"ln -s /usr/bin/pip3 /usr/local/bin/pip && ln -s /usr/bin/python3 /usr/bin/python"}]}},"platform":{"attributes":{"mountMode":"volume","restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"readinessTimeoutSeconds":120,"eventTimeout":"30s"}}}
24.05.16 07:41:56.073 (W)            nuctl.platform MaxWorkers is deprecated and will be removed in v1.15.x, use NumWorkers instead
24.05.16 07:41:56.073 (D)            nuctl.platform Enriching disable default http trigger flag {"functionName": "pth-facebookresearch-sam-vit-h", "disableDefaultHttpTrigger": false}
24.05.16 07:41:56.073 (D)            nuctl.platform Populating resources with default values {"defaultContainerResources": {"requests":{},"limits":{}}}
24.05.16 07:41:56.073 (D)            nuctl.platform Populated resources with default values {"resources": "&ResourceRequirements{Limits:ResourceList{nvidia.com/gpu: {{1 0} {<nil>} 1 DecimalSI},},Requests:ResourceList{cpu: {{25 -3} {<nil>} 25m DecimalSI},memory: {{1048576 0} {<nil>} 1Mi BinarySI},},Claims:[]ResourceClaim{},}"}
24.05.16 07:41:56.073 (D)     nuctl.platform.docker Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"/bin/cat /etc/nuclio/store/projects/nuclio/cvat.json\"","Stdout":"","Stderr":"","Env":null}}
24.05.16 07:41:56.200 (D)     nuctl.platform.docker Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"/bin/cat /etc/nuclio/store/functions/nuclio/pth-facebookresearch-sam-vit-h.json\"","Stdout":"","Stderr":"","Env":null}}
24.05.16 07:41:56.320 (I)                     nuctl Deploying function {"name": "pth-facebookresearch-sam-vit-h"}
24.05.16 07:41:56.320 (I)                     nuctl Building {"builderKind": "docker", "versionInfo": "Label: 1.13.0, Git commit: c4422eb772781fb50fbf017698aae96199d81388, OS: linux, Arch: amd64, Go version: go1.21.7", "name": "pth-facebookresearch-sam-vit-h"}
24.05.16 07:41:56.320 (D)                     nuctl Function configuration found in directory {"configFilePath": "pytorch/facebookresearch/sam/nuclio/function-gpu.yaml"}
24.05.16 07:41:56.321 (D)                     nuctl Created base temporary dir {"dir": "/tmp/nuclio-build-1029457993"}
24.05.16 07:41:56.321 (D)                     nuctl Created temporary dir {"dir": "/tmp/nuclio-build-1029457993/staging"}
24.05.16 07:41:56.321 (D)                     nuctl Created staging dir {"dir": "/tmp/nuclio-build-1029457993/staging"}
24.05.16 07:41:56.398 (D)                     nuctl Enriched configuration {"functionConfig": {"metadata":{"name":"pth-facebookresearch-sam-vit-h","namespace":"nuclio","labels":{"nuclio.io/project-name":"cvat"},"annotations":{"animated_gif":"https://raw.githubusercontent.com/opencv/cvat/develop/site/content/en/images/hrnet_example.gif","framework":"pytorch","help_message":"The interactor allows to get a mask of an object using at least one positive, and any negative points inside it","min_neg_points":"0","min_pos_points":"1","name":"Segment Anything","spec":"","type":"interactor","version":"2"}},"spec":{"description":"Interactive object segmentation with Segment-Anything","handler":"main:handler","runtime":"python:3.8","env":[{"name":"PYTHONPATH","value":"/opt/nuclio/sam"}],"resources":{"limits":{"nvidia.com/gpu":"1"},"requests":{"cpu":"25m","memory":"1Mi"}},"targetCPU":75,"triggers":{"myHttpTrigger":{"class":"","kind":"http","name":"myHttpTrigger","numWorkers":1,"workerAvailabilityTimeoutMilliseconds":10000,"attributes":{"maxRequestBodySize":33554432},"maxWorkers":1}},"build":{"path":"/home/gpuuser/cvat/serverless/pytorch/facebookresearch/sam/nuclio","functionConfigPath":"pytorch/facebookresearch/sam/nuclio/function-gpu.yaml","image":"cvat.pth.facebookresearch.sam.vit_h","baseImage":"ubuntu:22.04","directives":{"preCopy":[{"kind":"ENV","value":"NVIDIA_VISIBLE_DEVICES=all"},{"kind":"ENV","value":"NVIDIA_DRIVER_CAPABILITIES=compute,utility"},{"kind":"ENV","value":"DEBIAN_FRONTEND=noninteractive"},{"kind":"WORKDIR","value":"/opt/nuclio/sam"},{"kind":"RUN","value":"apt-get update && apt-get -y install curl git python3 python3-pip ffmpeg libsm6 libxext6"},{"kind":"RUN","value":"pip3 install torch torchvision torchaudio opencv-python pycocotools matplotlib onnxruntime onnx"},{"kind":"RUN","value":"pip3 install git+https://github.com/facebookresearch/segment-anything.git"},{"kind":"RUN","value":"curl -O https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth"},{"kind":"RUN","value":"ln -s /usr/bin/pip3 /usr/local/bin/pip && ln -s /usr/bin/python3 /usr/bin/python"}]},"codeEntryType":"image"},"platform":{"attributes":{"mountMode":"volume","restartPolicy":{"maximumRetryCount":3,"name":"always"}}},"readinessTimeoutSeconds":120,"securityContext":{},"disableDefaultHTTPTrigger":false,"eventTimeout":"30s"}}, "platform": "local", "dependantImagesRegistryURL": "", "outputImageFile": "", "processorImage": {}}
24.05.16 07:41:56.398 (D)                     nuctl Creating shadow function {"name": "pth-facebookresearch-sam-vit-h"}
24.05.16 07:41:56.398 (D)            nuctl.platform Populating resources with default values {"defaultContainerResources": {"requests":{},"limits":{}}}
24.05.16 07:41:56.398 (D)            nuctl.platform Populated resources with default values {"resources": "&ResourceRequirements{Limits:ResourceList{nvidia.com/gpu: {{1 0} {<nil>} 1 DecimalSI},},Requests:ResourceList{cpu: {{25 -3} {<nil>} 25m DecimalSI},memory: {{1048576 0} {<nil>} 1Mi BinarySI},},Claims:[]ResourceClaim{},}"}
24.05.16 07:41:56.398 (D)     nuctl.platform.docker Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"/bin/cat /etc/nuclio/store/projects/nuclio/cvat.json\"","Stdout":"","Stderr":"","Env":null}}
24.05.16 07:41:56.525 (D)               nuctl.store Writing file contents {"path": "/etc/nuclio/store/functions/nuclio/pth-facebookresearch-sam-vit-h.json", "contents": "{\"metadata\":{\"name\":\"pth-facebookresearch-sam-vit-h\",\"namespace\":\"nuclio\",\"labels\":{\"nuclio.io/project-name\":\"cvat\"},\"annotations\":{\"animated_gif\":\"https://raw.githubusercontent.com/opencv/cvat/develop/site/content/en/images/hrnet_example.gif\",\"framework\":\"pytorch\",\"help_message\":\"The interactor allows to get a mask of an object using at least one positive, and any negative points inside it\",\"min_neg_points\":\"0\",\"min_pos_points\":\"1\",\"name\":\"Segment Anything\",\"spec\":\"\",\"type\":\"interactor\",\"version\":\"2\"}},\"spec\":{\"description\":\"Interactive object segmentation with Segment-Anything\",\"handler\":\"main:handler\",\"runtime\":\"python:3.8\",\"env\":[{\"name\":\"PYTHONPATH\",\"value\":\"/opt/nuclio/sam\"}],\"resources\":{\"limits\":{\"nvidia.com/gpu\":\"1\"},\"requests\":{\"cpu\":\"25m\",\"memory\":\"1Mi\"}},\"targetCPU\":75,\"triggers\":{\"myHttpTrigger\":{\"class\":\"\",\"kind\":\"http\",\"name\":\"myHttpTrigger\",\"numWorkers\":1,\"workerAvailabilityTimeoutMilliseconds\":10000,\"attributes\":{\"maxRequestBodySize\":33554432},\"maxWorkers\":1}},\"build\":{\"functionConfigPath\":\"pytorch/facebookresearch/sam/nuclio/function-gpu.yaml\",\"image\":\"cvat.pth.facebookresearch.sam.vit_h\",\"baseImage\":\"ubuntu:22.04\",\"directives\":{\"preCopy\":[{\"kind\":\"ENV\",\"value\":\"NVIDIA_VISIBLE_DEVICES=all\"},{\"kind\":\"ENV\",\"value\":\"NVIDIA_DRIVER_CAPABILITIES=compute,utility\"},{\"kind\":\"ENV\",\"value\":\"DEBIAN_FRONTEND=noninteractive\"},{\"kind\":\"WORKDIR\",\"value\":\"/opt/nuclio/sam\"},{\"kind\":\"RUN\",\"value\":\"apt-get update \\u0026\\u0026 apt-get -y install curl git python3 python3-pip ffmpeg libsm6 libxext6\"},{\"kind\":\"RUN\",\"value\":\"pip3 install torch torchvision torchaudio opencv-python pycocotools matplotlib onnxruntime onnx\"},{\"kind\":\"RUN\",\"value\":\"pip3 install git+https://github.com/facebookresearch/segment-anything.git\"},{\"kind\":\"RUN\",\"value\":\"curl -O https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth\"},{\"kind\":\"RUN\",\"value\":\"ln -s /usr/bin/pip3 /usr/local/bin/pip \\u0026\\u0026 ln -s /usr/bin/python3 /usr/bin/python\"}]},\"codeEntryType\":\"image\"},\"platform\":{\"attributes\":{\"mountMode\":\"volume\",\"restartPolicy\":{\"maximumRetryCount\":3,\"name\":\"always\"}}},\"readinessTimeoutSeconds\":120,\"securityContext\":{},\"disableDefaultHTTPTrigger\":false,\"eventTimeout\":\"30s\"},\"status\":{\"state\":\"building\"}}"}
24.05.16 07:41:56.525 (D)     nuctl.platform.docker Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"mkdir -p /etc/nuclio/store/functions/nuclio && /bin/printenv NUCLIO_CONTENTS > /etc/nuclio/store/functions/nuclio/pth-facebookresearch-sam-vit-h.json\"","Stdout":"","Stderr":"","Env":{"NUCLIO_CONTENTS":"eyJtZXRhZGF0YSI6eyJuYW1lIjoicHRoLWZhY2Vib29rcmVzZWFyY2gtc2FtLXZpdC1oIiwibmFtZXNwYWNlIjoibnVjbGlvIiwibGFiZWxzIjp7Im51Y2xpby5pby9wcm9qZWN0LW5hbWUiOiJjdmF0In0sImFubm90YXRpb25zIjp7ImFuaW1hdGVkX2dpZiI6Imh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9vcGVuY3YvY3ZhdC9kZXZlbG9wL3NpdGUvY29udGVudC9lbi9pbWFnZXMvaHJuZXRfZXhhbXBsZS5naWYiLCJmcmFtZXdvcmsiOiJweXRvcmNoIiwiaGVscF9tZXNzYWdlIjoiVGhlIGludGVyYWN0b3IgYWxsb3dzIHRvIGdldCBhIG1hc2sgb2YgYW4gb2JqZWN0IHVzaW5nIGF0IGxlYXN0IG9uZSBwb3NpdGl2ZSwgYW5kIGFueSBuZWdhdGl2ZSBwb2ludHMgaW5zaWRlIGl0IiwibWluX25lZ19wb2ludHMiOiIwIiwibWluX3Bvc19wb2ludHMiOiIxIiwibmFtZSI6IlNlZ21lbnQgQW55dGhpbmciLCJzcGVjIjoiIiwidHlwZSI6ImludGVyYWN0b3IiLCJ2ZXJzaW9uIjoiMiJ9fSwic3BlYyI6eyJkZXNjcmlwdGlvbiI6IkludGVyYWN0aXZlIG9iamVjdCBzZWdtZW50YXRpb24gd2l0aCBTZWdtZW50LUFueXRoaW5nIiwiaGFuZGxlciI6Im1haW46aGFuZGxlciIsInJ1bnRpbWUiOiJweXRob246My44IiwiZW52IjpbeyJuYW1lIjoiUFlUSE9OUEFUSCIsInZhbHVlIjoiL29wdC9udWNsaW8vc2FtIn1dLCJyZXNvdXJjZXMiOnsibGltaXRzIjp7Im52aWRpYS5jb20vZ3B1IjoiMSJ9LCJyZXF1ZXN0cyI6eyJjcHUiOiIyNW0iLCJtZW1vcnkiOiIxTWkifX0sInRhcmdldENQVSI6NzUsInRyaWdnZXJzIjp7Im15SHR0cFRyaWdnZXIiOnsiY2xhc3MiOiIiLCJraW5kIjoiaHR0cCIsIm5hbWUiOiJteUh0dHBUcmlnZ2VyIiwibnVtV29ya2VycyI6MSwid29ya2VyQXZhaWxhYmlsaXR5VGltZW91dE1pbGxpc2Vjb25kcyI6MTAwMDAsImF0dHJpYnV0ZXMiOnsibWF4UmVxdWVzdEJvZHlTaXplIjozMzU1NDQzMn0sIm1heFdvcmtlcnMiOjF9fSwiYnVpbGQiOnsiZnVuY3Rpb25Db25maWdQYXRoIjoicHl0b3JjaC9mYWNlYm9va3Jlc2VhcmNoL3NhbS9udWNsaW8vZnVuY3Rpb24tZ3B1LnlhbWwiLCJpbWFnZSI6ImN2YXQucHRoLmZhY2Vib29rcmVzZWFyY2guc2FtLnZpdF9oIiwiYmFzZUltYWdlIjoidWJ1bnR1OjIyLjA0IiwiZGlyZWN0aXZlcyI6eyJwcmVDb3B5IjpbeyJraW5kIjoiRU5WIiwidmFsdWUiOiJOVklESUFfVklTSUJMRV9ERVZJQ0VTPWFsbCJ9LHsia2luZCI6IkVOViIsInZhbHVlIjoiTlZJRElBX0RSSVZFUl9DQVBBQklMSVRJRVM9Y29tcHV0ZSx1dGlsaXR5In0seyJraW5kIjoiRU5WIiwidmFsdWUiOiJERUJJQU5fRlJPTlRFTkQ9bm9uaW50ZXJhY3RpdmUifSx7ImtpbmQiOiJXT1JLRElSIiwidmFsdWUiOiIvb3B0L251Y2xpby9zYW0ifSx7ImtpbmQiOiJSVU4iLCJ2YWx1ZSI6ImFwdC1nZXQgdXBkYXRlIFx1MDAyNlx1MDAyNiBhcHQtZ2V0IC15IGluc3RhbGwgY3VybCBnaXQgcHl0aG9uMyBweXRob24zLXBpcCBmZm1wZWcgbGlic202IGxpYnhleHQ2In0seyJraW5kIjoiUlVOIiwidmFsdWUiOiJwaXAzIGluc3RhbGwgdG9yY2ggdG9yY2h2aXNpb24gdG9yY2hhdWRpbyBvcGVuY3YtcHl0aG9uIHB5Y29jb3Rvb2xzIG1hdHBsb3RsaWIgb25ueHJ1bnRpbWUgb25ueCJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoicGlwMyBpbnN0YWxsIGdpdCtodHRwczovL2dpdGh1Yi5jb20vZmFjZWJvb2tyZXNlYXJjaC9zZWdtZW50LWFueXRoaW5nLmdpdCJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoiY3VybCAtTyBodHRwczovL2RsLmZiYWlwdWJsaWNmaWxlcy5jb20vc2VnbWVudF9hbnl0aGluZy9zYW1fdml0X2hfNGI4OTM5LnB0aCJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoibG4gLXMgL3Vzci9iaW4vcGlwMyAvdXNyL2xvY2FsL2Jpbi9waXAgXHUwMDI2XHUwMDI2IGxuIC1zIC91c3IvYmluL3B5dGhvbjMgL3Vzci9iaW4vcHl0aG9uIn1dfSwiY29kZUVudHJ5VHlwZSI6ImltYWdlIn0sInBsYXRmb3JtIjp7ImF0dHJpYnV0ZXMiOnsibW91bnRNb2RlIjoidm9sdW1lIiwicmVzdGFydFBvbGljeSI6eyJtYXhpbXVtUmV0cnlDb3VudCI6MywibmFtZSI6ImFsd2F5cyJ9fX0sInJlYWRpbmVzc1RpbWVvdXRTZWNvbmRzIjoxMjAsInNlY3VyaXR5Q29udGV4dCI6e30sImRpc2FibGVEZWZhdWx0SFRUUFRyaWdnZXIiOmZhbHNlLCJldmVudFRpbWVvdXQiOiIzMHMifSwic3RhdHVzIjp7InN0YXRlIjoiYnVpbGRpbmcifX0="}}}
24.05.16 07:41:56.634 (I)                     nuctl Staging files and preparing base images
24.05.16 07:41:56.634 (D)                     nuctl Runtime provided handler objects to staging dir {"handlerDirObjectPaths": ["/home/gpuuser/cvat/serverless/pytorch/facebookresearch/sam/nuclio"], "handlerDirInStaging": "/tmp/nuclio-build-1029457993/staging/handler", "handlerDirIncludingSubpath": "/tmp/nuclio-build-1029457993/staging/handler"}
24.05.16 07:41:56.634 (W)                     nuctl Python 3.8 runtime is deprecated and will soon not be supported. Migrate your code and use Python 3.9 runtime (`python:3.9`) or higher
24.05.16 07:41:56.634 (W)                     nuctl Using user provided base image, runtime interpreter version is provided by the base image {"baseImage": "ubuntu:22.04"}
24.05.16 07:41:56.634 (D)                     nuctl Created processor Dockerfile {"dockerfileInfo": "# Multistage builds\n\n\n\n# From the base image\nFROM ubuntu:22.04\n\n\n\n# Old(er) Docker support - must use all build args\nARG NUCLIO_LABEL\nARG NUCLIO_ARCH\nARG NUCLIO_BUILD_LOCAL_HANDLER_DIR\n\n\n# Run the pre-copy directives\n\nENV NVIDIA_VISIBLE_DEVICES=all\n\nENV NVIDIA_DRIVER_CAPABILITIES=compute,utility\n\nENV DEBIAN_FRONTEND=noninteractive\n\nWORKDIR /opt/nuclio/sam\n\nRUN apt-get update && apt-get -y install curl git python3 python3-pip ffmpeg libsm6 libxext6\n\nRUN pip3 install torch torchvision torchaudio opencv-python pycocotools matplotlib onnxruntime onnx\n\nRUN pip3 install git+https://github.com/facebookresearch/segment-anything.git\n\nRUN curl -O https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth\n\nRUN ln -s /usr/bin/pip3 /usr/local/bin/pip && ln -s /usr/bin/python3 /usr/bin/python\n\n\n\n# Copy required objects from the suppliers\n\nCOPY artifacts/processor /usr/local/bin/processor\n\nCOPY artifacts/py /opt/nuclio/\n\nCOPY artifacts/py3.8-whl /opt/nuclio/whl\n\nCOPY artifacts/uhttpc /usr/local/bin/uhttpc\n\n\n\nCOPY handler /opt/nuclio\n\n\n\n# Readiness probe\nHEALTHCHECK --interval=1s --timeout=3s CMD /usr/local/bin/uhttpc --url http://127.0.0.1:8082/ready || exit 1\n\n\n# Run the post-copy directives\n\nRUN python /opt/nuclio/whl/$(basename /opt/nuclio/whl/pip-*.whl)/pip install pip --no-index --find-links /opt/nuclio/whl && python -m pip install nuclio-sdk msgpack --no-index --find-links /opt/nuclio/whl\n\n\n# Run processor with configuration and platform configuration\nCMD [ \"processor\" ]\n", "baseImageRegistry": "", "onbuildImageRegistry": "quay.io"}
24.05.16 07:41:56.635 (I)                     nuctl Building processor image {"registryURL": "", "taggedImageName": "cvat.pth.facebookresearch.sam.vit_h:latest"}
24.05.16 07:41:56.635 (I)     nuctl.platform.docker Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.13.0-amd64"}
24.05.16 07:41:56.635 (D) tl.platform.docker.runner Executing {"command": "docker pull quay.io/nuclio/handler-builder-python-onbuild:1.13.0-amd64"}
24.05.16 07:41:57.812 (D) tl.platform.docker.runner Command executed successfully {"output": "1.13.0-amd64: Pulling from nuclio/handler-builder-python-onbuild\nDigest: sha256:4d73b690f3be017926012a4b1d06242dd5b6274bc5ffbb847e9b881ac57983ed\nStatus: Image is up to date for quay.io/nuclio/handler-builder-python-onbuild:1.13.0-amd64\nquay.io/nuclio/handler-builder-python-onbuild:1.13.0-amd64\n", "stderr": "", "exitCode": 0}
24.05.16 07:41:57.812 (D)            nuctl.platform Generated onbuild Dockerfile {"contents": "FROM quay.io/nuclio/handler-builder-python-onbuild:1.13.0-amd64\nARG NUCLIO_LABEL\nARG NUCLIO_ARCH\n"}
24.05.16 07:41:57.812 (D)     nuctl.platform.docker Building image {"buildOptions": {"Image":"nuclio-onbuild-cp2rhh8orl71q4onc9p0","ContextDir":"/tmp/nuclio-build-1029457993/staging","DockerfilePath":"/tmp/nuclio-build-1029457993/staging/Dockerfile.onbuild","NoCache":false,"Pull":false,"BuildArgs":{"NUCLIO_ARCH":"amd64","NUCLIO_BUILD_LOCAL_HANDLER_DIR":"handler","NUCLIO_LABEL":"1.13.0"},"BuildFlags":null}}
24.05.16 07:41:57.812 (D) tl.platform.docker.runner Executing {"command": "docker build --network host --force-rm -t nuclio-onbuild-cp2rhh8orl71q4onc9p0 -f /tmp/nuclio-build-1029457993/staging/Dockerfile.onbuild   --build-arg NUCLIO_LABEL=1.13.0 --build-arg NUCLIO_ARCH=amd64 --build-arg NUCLIO_BUILD_LOCAL_HANDLER_DIR=handler  ."}
24.05.16 07:41:58.006 (D) tl.platform.docker.runner Failed to execute command {"output": "", "stderr": "ERROR: could not find /tmp/nuclio-build-1029457993/staging: stat /tmp/nuclio-build-1029457993/staging: no such file or directory\n", "exitCode": 1, "err": "exit status 1"}
24.05.16 07:41:58.006 (W)     nuctl.platform.docker Docker command outputted to stderr - this may result in errors {"workingDir": "/tmp/nuclio-build-1029457993/staging", "cmd": "docker build --network host --force-rm -t nuclio-onbuild-cp2rhh8orl71q4onc9p0 -f /tmp/nuclio-build-1029457993/staging/Dockerfile.onbuild   --build-arg NUCLIO_LABEL=1.13.0 --build-arg NUCLIO_ARCH=amd64 --build-arg NUCLIO_BUILD_LOCAL_HANDLER_DIR=handler  .", "stderr": "ERROR: could not find /tmp/nuclio-build-1029457993/staging: stat /tmp/nuclio-build-1029457993/staging: no such file or directory\n"}
24.05.16 07:41:58.007 (D)                     nuctl Successfully cleaned up temporary dir {"dir": "/tmp/nuclio-build-1029457993"}
24.05.16 07:41:58.007 (W)                     nuctl Failed to create a function; setting the function status {"err": "Failed to build processor image", "errVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/shellrunner.go:114\n\nCall stack:\nstdout:\n\nstderr:\nERROR: could not find /tmp/nuclio-build-1029457993/staging: stat /tmp/nuclio-build-1029457993/staging: no such file or directory\n\n    /nuclio/pkg/cmdrunner/shellrunner.go:114\nFailed to build\n    /nuclio/pkg/dockerclient/shell.go:119\nFailed to build onbuild image\n    .../pkg/containerimagebuilderpusher/docker.go:257\nFailed to copy objects from onbuild\n    .../pkg/containerimagebuilderpusher/docker.go:217\nFailed to build image artifacts\n    .../pkg/containerimagebuilderpusher/docker.go:66\nFailed to build processor image\n    /nuclio/pkg/processor/build/builder.go:267\nFailed to build processor image"}
24.05.16 07:41:58.007 (D)               nuctl.store Writing file contents {"path": "/etc/nuclio/store/functions/nuclio/pth-facebookresearch-sam-vit-h.json", "contents": "{\"metadata\":{\"name\":\"pth-facebookresearch-sam-vit-h\",\"namespace\":\"nuclio\",\"labels\":{\"nuclio.io/project-name\":\"cvat\"},\"annotations\":{\"animated_gif\":\"https://raw.githubusercontent.com/opencv/cvat/develop/site/content/en/images/hrnet_example.gif\",\"framework\":\"pytorch\",\"help_message\":\"The interactor allows to get a mask of an object using at least one positive, and any negative points inside it\",\"min_neg_points\":\"0\",\"min_pos_points\":\"1\",\"name\":\"Segment Anything\",\"spec\":\"\",\"type\":\"interactor\",\"version\":\"2\"}},\"spec\":{\"description\":\"Interactive object segmentation with Segment-Anything\",\"handler\":\"main:handler\",\"runtime\":\"python:3.8\",\"env\":[{\"name\":\"PYTHONPATH\",\"value\":\"/opt/nuclio/sam\"}],\"resources\":{\"limits\":{\"nvidia.com/gpu\":\"1\"},\"requests\":{\"cpu\":\"25m\",\"memory\":\"1Mi\"}},\"targetCPU\":75,\"triggers\":{\"myHttpTrigger\":{\"class\":\"\",\"kind\":\"http\",\"name\":\"myHttpTrigger\",\"numWorkers\":1,\"workerAvailabilityTimeoutMilliseconds\":10000,\"attributes\":{\"maxRequestBodySize\":33554432},\"maxWorkers\":1}},\"build\":{\"functionConfigPath\":\"pytorch/facebookresearch/sam/nuclio/function-gpu.yaml\",\"image\":\"cvat.pth.facebookresearch.sam.vit_h\",\"baseImage\":\"ubuntu:22.04\",\"directives\":{\"preCopy\":[{\"kind\":\"ENV\",\"value\":\"NVIDIA_VISIBLE_DEVICES=all\"},{\"kind\":\"ENV\",\"value\":\"NVIDIA_DRIVER_CAPABILITIES=compute,utility\"},{\"kind\":\"ENV\",\"value\":\"DEBIAN_FRONTEND=noninteractive\"},{\"kind\":\"WORKDIR\",\"value\":\"/opt/nuclio/sam\"},{\"kind\":\"RUN\",\"value\":\"apt-get update \\u0026\\u0026 apt-get -y install curl git python3 python3-pip ffmpeg libsm6 libxext6\"},{\"kind\":\"RUN\",\"value\":\"pip3 install torch torchvision torchaudio opencv-python pycocotools matplotlib onnxruntime onnx\"},{\"kind\":\"RUN\",\"value\":\"pip3 install git+https://github.com/facebookresearch/segment-anything.git\"},{\"kind\":\"RUN\",\"value\":\"curl -O https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth\"},{\"kind\":\"RUN\",\"value\":\"ln -s /usr/bin/pip3 /usr/local/bin/pip \\u0026\\u0026 ln -s /usr/bin/python3 /usr/bin/python\"}]},\"codeEntryType\":\"image\"},\"platform\":{\"attributes\":{\"mountMode\":\"volume\",\"restartPolicy\":{\"maximumRetryCount\":3,\"name\":\"always\"}}},\"readinessTimeoutSeconds\":120,\"securityContext\":{},\"disableDefaultHTTPTrigger\":false,\"eventTimeout\":\"30s\"},\"status\":{\"state\":\"error\",\"message\":\"\\nError - exit status 1\\n    /nuclio/pkg/cmdrunner/shellrunner.go:114\\n\\nCall stack:\\nstdout:\\n\\nstderr:\\nERROR: could not find /tmp/nuclio-build-1029457993/staging: stat /tmp/nuclio-build-1029457993/staging: no such file or directory\\n\\n    /nuclio/pkg/cmdrunner/shellrunner.go:114\\nFailed to build\\n    /nuclio/pkg/dockerclient/shell.go:119\\nFailed to build onbuild image\\n    .../pkg/containerimagebuilderpusher/docker.go:257\\nFailed to copy objects from onbuild\\n    .../pkg/containerimagebuilderpusher/docker.go:217\\nFailed to build image artifacts\\n    .../pkg/containerimagebuilderpusher/docker.go:66\\nFailed to build processor image\\n    /nuclio/pkg/processor/build/builder.go:267\\n\"}}"}
24.05.16 07:41:58.007 (D)     nuctl.platform.docker Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"mkdir -p /etc/nuclio/store/functions/nuclio && /bin/printenv NUCLIO_CONTENTS > /etc/nuclio/store/functions/nuclio/pth-facebookresearch-sam-vit-h.json\"","Stdout":"","Stderr":"","Env":{"NUCLIO_CONTENTS":"eyJtZXRhZGF0YSI6eyJuYW1lIjoicHRoLWZhY2Vib29rcmVzZWFyY2gtc2FtLXZpdC1oIiwibmFtZXNwYWNlIjoibnVjbGlvIiwibGFiZWxzIjp7Im51Y2xpby5pby9wcm9qZWN0LW5hbWUiOiJjdmF0In0sImFubm90YXRpb25zIjp7ImFuaW1hdGVkX2dpZiI6Imh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9vcGVuY3YvY3ZhdC9kZXZlbG9wL3NpdGUvY29udGVudC9lbi9pbWFnZXMvaHJuZXRfZXhhbXBsZS5naWYiLCJmcmFtZXdvcmsiOiJweXRvcmNoIiwiaGVscF9tZXNzYWdlIjoiVGhlIGludGVyYWN0b3IgYWxsb3dzIHRvIGdldCBhIG1hc2sgb2YgYW4gb2JqZWN0IHVzaW5nIGF0IGxlYXN0IG9uZSBwb3NpdGl2ZSwgYW5kIGFueSBuZWdhdGl2ZSBwb2ludHMgaW5zaWRlIGl0IiwibWluX25lZ19wb2ludHMiOiIwIiwibWluX3Bvc19wb2ludHMiOiIxIiwibmFtZSI6IlNlZ21lbnQgQW55dGhpbmciLCJzcGVjIjoiIiwidHlwZSI6ImludGVyYWN0b3IiLCJ2ZXJzaW9uIjoiMiJ9fSwic3BlYyI6eyJkZXNjcmlwdGlvbiI6IkludGVyYWN0aXZlIG9iamVjdCBzZWdtZW50YXRpb24gd2l0aCBTZWdtZW50LUFueXRoaW5nIiwiaGFuZGxlciI6Im1haW46aGFuZGxlciIsInJ1bnRpbWUiOiJweXRob246My44IiwiZW52IjpbeyJuYW1lIjoiUFlUSE9OUEFUSCIsInZhbHVlIjoiL29wdC9udWNsaW8vc2FtIn1dLCJyZXNvdXJjZXMiOnsibGltaXRzIjp7Im52aWRpYS5jb20vZ3B1IjoiMSJ9LCJyZXF1ZXN0cyI6eyJjcHUiOiIyNW0iLCJtZW1vcnkiOiIxTWkifX0sInRhcmdldENQVSI6NzUsInRyaWdnZXJzIjp7Im15SHR0cFRyaWdnZXIiOnsiY2xhc3MiOiIiLCJraW5kIjoiaHR0cCIsIm5hbWUiOiJteUh0dHBUcmlnZ2VyIiwibnVtV29ya2VycyI6MSwid29ya2VyQXZhaWxhYmlsaXR5VGltZW91dE1pbGxpc2Vjb25kcyI6MTAwMDAsImF0dHJpYnV0ZXMiOnsibWF4UmVxdWVzdEJvZHlTaXplIjozMzU1NDQzMn0sIm1heFdvcmtlcnMiOjF9fSwiYnVpbGQiOnsiZnVuY3Rpb25Db25maWdQYXRoIjoicHl0b3JjaC9mYWNlYm9va3Jlc2VhcmNoL3NhbS9udWNsaW8vZnVuY3Rpb24tZ3B1LnlhbWwiLCJpbWFnZSI6ImN2YXQucHRoLmZhY2Vib29rcmVzZWFyY2guc2FtLnZpdF9oIiwiYmFzZUltYWdlIjoidWJ1bnR1OjIyLjA0IiwiZGlyZWN0aXZlcyI6eyJwcmVDb3B5IjpbeyJraW5kIjoiRU5WIiwidmFsdWUiOiJOVklESUFfVklTSUJMRV9ERVZJQ0VTPWFsbCJ9LHsia2luZCI6IkVOViIsInZhbHVlIjoiTlZJRElBX0RSSVZFUl9DQVBBQklMSVRJRVM9Y29tcHV0ZSx1dGlsaXR5In0seyJraW5kIjoiRU5WIiwidmFsdWUiOiJERUJJQU5fRlJPTlRFTkQ9bm9uaW50ZXJhY3RpdmUifSx7ImtpbmQiOiJXT1JLRElSIiwidmFsdWUiOiIvb3B0L251Y2xpby9zYW0ifSx7ImtpbmQiOiJSVU4iLCJ2YWx1ZSI6ImFwdC1nZXQgdXBkYXRlIFx1MDAyNlx1MDAyNiBhcHQtZ2V0IC15IGluc3RhbGwgY3VybCBnaXQgcHl0aG9uMyBweXRob24zLXBpcCBmZm1wZWcgbGlic202IGxpYnhleHQ2In0seyJraW5kIjoiUlVOIiwidmFsdWUiOiJwaXAzIGluc3RhbGwgdG9yY2ggdG9yY2h2aXNpb24gdG9yY2hhdWRpbyBvcGVuY3YtcHl0aG9uIHB5Y29jb3Rvb2xzIG1hdHBsb3RsaWIgb25ueHJ1bnRpbWUgb25ueCJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoicGlwMyBpbnN0YWxsIGdpdCtodHRwczovL2dpdGh1Yi5jb20vZmFjZWJvb2tyZXNlYXJjaC9zZWdtZW50LWFueXRoaW5nLmdpdCJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoiY3VybCAtTyBodHRwczovL2RsLmZiYWlwdWJsaWNmaWxlcy5jb20vc2VnbWVudF9hbnl0aGluZy9zYW1fdml0X2hfNGI4OTM5LnB0aCJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoibG4gLXMgL3Vzci9iaW4vcGlwMyAvdXNyL2xvY2FsL2Jpbi9waXAgXHUwMDI2XHUwMDI2IGxuIC1zIC91c3IvYmluL3B5dGhvbjMgL3Vzci9iaW4vcHl0aG9uIn1dfSwiY29kZUVudHJ5VHlwZSI6ImltYWdlIn0sInBsYXRmb3JtIjp7ImF0dHJpYnV0ZXMiOnsibW91bnRNb2RlIjoidm9sdW1lIiwicmVzdGFydFBvbGljeSI6eyJtYXhpbXVtUmV0cnlDb3VudCI6MywibmFtZSI6ImFsd2F5cyJ9fX0sInJlYWRpbmVzc1RpbWVvdXRTZWNvbmRzIjoxMjAsInNlY3VyaXR5Q29udGV4dCI6e30sImRpc2FibGVEZWZhdWx0SFRUUFRyaWdnZXIiOmZhbHNlLCJldmVudFRpbWVvdXQiOiIzMHMifSwic3RhdHVzIjp7InN0YXRlIjoiZXJyb3IiLCJtZXNzYWdlIjoiXG5FcnJvciAtIGV4aXQgc3RhdHVzIDFcbiAgICAvbnVjbGlvL3BrZy9jbWRydW5uZXIvc2hlbGxydW5uZXIuZ286MTE0XG5cbkNhbGwgc3RhY2s6XG5zdGRvdXQ6XG5cbnN0ZGVycjpcbkVSUk9SOiBjb3VsZCBub3QgZmluZCAvdG1wL251Y2xpby1idWlsZC0xMDI5NDU3OTkzL3N0YWdpbmc6IHN0YXQgL3RtcC9udWNsaW8tYnVpbGQtMTAyOTQ1Nzk5My9zdGFnaW5nOiBubyBzdWNoIGZpbGUgb3IgZGlyZWN0b3J5XG5cbiAgICAvbnVjbGlvL3BrZy9jbWRydW5uZXIvc2hlbGxydW5uZXIuZ286MTE0XG5GYWlsZWQgdG8gYnVpbGRcbiAgICAvbnVjbGlvL3BrZy9kb2NrZXJjbGllbnQvc2hlbGwuZ286MTE5XG5GYWlsZWQgdG8gYnVpbGQgb25idWlsZCBpbWFnZVxuICAgIC4uLi9wa2cvY29udGFpbmVyaW1hZ2VidWlsZGVycHVzaGVyL2RvY2tlci5nbzoyNTdcbkZhaWxlZCB0byBjb3B5IG9iamVjdHMgZnJvbSBvbmJ1aWxkXG4gICAgLi4uL3BrZy9jb250YWluZXJpbWFnZWJ1aWxkZXJwdXNoZXIvZG9ja2VyLmdvOjIxN1xuRmFpbGVkIHRvIGJ1aWxkIGltYWdlIGFydGlmYWN0c1xuICAgIC4uLi9wa2cvY29udGFpbmVyaW1hZ2VidWlsZGVycHVzaGVyL2RvY2tlci5nbzo2NlxuRmFpbGVkIHRvIGJ1aWxkIHByb2Nlc3NvciBpbWFnZVxuICAgIC9udWNsaW8vcGtnL3Byb2Nlc3Nvci9idWlsZC9idWlsZGVyLmdvOjI2N1xuIn19"}}}
24.05.16 07:41:58.132 (D)                     nuctl Saving deployment logs
24.05.16 07:41:58.132 (D)     nuctl.platform.docker Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"/bin/cat /etc/nuclio/store/functions/nuclio/pth-facebookresearch-sam-vit-h.json\"","Stdout":"","Stderr":"","Env":null}}
24.05.16 07:41:58.247 (D)               nuctl.store Writing file contents {"path": "/etc/nuclio/store/functions/nuclio/pth-facebookresearch-sam-vit-h.json", "contents": "{\"metadata\":{\"name\":\"pth-facebookresearch-sam-vit-h\",\"namespace\":\"nuclio\",\"labels\":{\"nuclio.io/project-name\":\"cvat\"},\"annotations\":{\"animated_gif\":\"https://raw.githubusercontent.com/opencv/cvat/develop/site/content/en/images/hrnet_example.gif\",\"framework\":\"pytorch\",\"help_message\":\"The interactor allows to get a mask of an object using at least one positive, and any negative points inside it\",\"min_neg_points\":\"0\",\"min_pos_points\":\"1\",\"name\":\"Segment Anything\",\"spec\":\"\",\"type\":\"interactor\",\"version\":\"2\"}},\"spec\":{\"description\":\"Interactive object segmentation with Segment-Anything\",\"handler\":\"main:handler\",\"runtime\":\"python:3.8\",\"env\":[{\"name\":\"PYTHONPATH\",\"value\":\"/opt/nuclio/sam\"}],\"resources\":{\"limits\":{\"nvidia.com/gpu\":\"1\"},\"requests\":{\"cpu\":\"25m\",\"memory\":\"1Mi\"}},\"targetCPU\":75,\"triggers\":{\"myHttpTrigger\":{\"class\":\"\",\"kind\":\"http\",\"name\":\"myHttpTrigger\",\"numWorkers\":1,\"workerAvailabilityTimeoutMilliseconds\":10000,\"attributes\":{\"maxRequestBodySize\":33554432},\"maxWorkers\":1}},\"build\":{\"functionConfigPath\":\"pytorch/facebookresearch/sam/nuclio/function-gpu.yaml\",\"image\":\"cvat.pth.facebookresearch.sam.vit_h\",\"baseImage\":\"ubuntu:22.04\",\"directives\":{\"preCopy\":[{\"kind\":\"ENV\",\"value\":\"NVIDIA_VISIBLE_DEVICES=all\"},{\"kind\":\"ENV\",\"value\":\"NVIDIA_DRIVER_CAPABILITIES=compute,utility\"},{\"kind\":\"ENV\",\"value\":\"DEBIAN_FRONTEND=noninteractive\"},{\"kind\":\"WORKDIR\",\"value\":\"/opt/nuclio/sam\"},{\"kind\":\"RUN\",\"value\":\"apt-get update \\u0026\\u0026 apt-get -y install curl git python3 python3-pip ffmpeg libsm6 libxext6\"},{\"kind\":\"RUN\",\"value\":\"pip3 install torch torchvision torchaudio opencv-python pycocotools matplotlib onnxruntime onnx\"},{\"kind\":\"RUN\",\"value\":\"pip3 install git+https://github.com/facebookresearch/segment-anything.git\"},{\"kind\":\"RUN\",\"value\":\"curl -O https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth\"},{\"kind\":\"RUN\",\"value\":\"ln -s /usr/bin/pip3 /usr/local/bin/pip \\u0026\\u0026 ln -s /usr/bin/python3 /usr/bin/python\"}]},\"codeEntryType\":\"image\"},\"platform\":{\"attributes\":{\"mountMode\":\"volume\",\"restartPolicy\":{\"maximumRetryCount\":3,\"name\":\"always\"}}},\"readinessTimeoutSeconds\":120,\"securityContext\":{},\"disableDefaultHTTPTrigger\":false,\"eventTimeout\":\"30s\"},\"status\":{\"state\":\"error\",\"message\":\"\\nError - exit status 1\\n    /nuclio/pkg/cmdrunner/shellrunner.go:114\\n\\nCall stack:\\nstdout:\\n\\nstderr:\\nERROR: could not find /tmp/nuclio-build-1029457993/staging: stat /tmp/nuclio-build-1029457993/staging: no such file or directory\\n\\n    /nuclio/pkg/cmdrunner/shellrunner.go:114\\nFailed to build\\n    /nuclio/pkg/dockerclient/shell.go:119\\nFailed to build onbuild image\\n    .../pkg/containerimagebuilderpusher/docker.go:257\\nFailed to copy objects from onbuild\\n    .../pkg/containerimagebuilderpusher/docker.go:217\\nFailed to build image artifacts\\n    .../pkg/containerimagebuilderpusher/docker.go:66\\nFailed to build processor image\\n    /nuclio/pkg/processor/build/builder.go:267\\n\",\"logs\":[{\"level\":\"info\",\"message\":\"Deploying function\",\"name\":\"pth-facebookresearch-sam-vit-h\",\"time\":1715845316320.238},{\"builderKind\":\"docker\",\"level\":\"info\",\"message\":\"Building\",\"name\":\"pth-facebookresearch-sam-vit-h\",\"time\":1715845316320.2827,\"versionInfo\":\"Label: 1.13.0, Git commit: c4422eb772781fb50fbf017698aae96199d81388, OS: linux, Arch: amd64, Go version: go1.21.7\"},{\"level\":\"info\",\"message\":\"Staging files and preparing base images\",\"name\":\"deployer\",\"time\":1715845316634.4033},{\"level\":\"warn\",\"message\":\"Python 3.8 runtime is deprecated and will soon not be supported. Migrate your code and use Python 3.9 runtime (`python:3.9`) or higher\",\"name\":\"deployer\",\"time\":1715845316634.8066},{\"baseImage\":\"ubuntu:22.04\",\"level\":\"warn\",\"message\":\"Using user provided base image, runtime interpreter version is provided by the base image\",\"name\":\"deployer\",\"time\":1715845316634.828},{\"level\":\"info\",\"message\":\"Building processor image\",\"name\":\"deployer\",\"registryURL\":\"\",\"taggedImageName\":\"cvat.pth.facebookresearch.sam.vit_h:latest\",\"time\":1715845316635.0562},{\"err\":\"Failed to build processor image\",\"errVerbose\":\"\\nError - exit status 1\\n    /nuclio/pkg/cmdrunner/shellrunner.go:114\\n\\nCall stack:\\nstdout:\\n\\nstderr:\\nERROR: could not find /tmp/nuclio-build-1029457993/staging: stat /tmp/nuclio-build-1029457993/staging: no such file or directory\\n\\n    /nuclio/pkg/cmdrunner/shellrunner.go:114\\nFailed to build\\n    /nuclio/pkg/dockerclient/shell.go:119\\nFailed to build onbuild image\\n    .../pkg/containerimagebuilderpusher/docker.go:257\\nFailed to copy objects from onbuild\\n    .../pkg/containerimagebuilderpusher/docker.go:217\\nFailed to build image artifacts\\n    .../pkg/containerimagebuilderpusher/docker.go:66\\nFailed to build processor image\\n    /nuclio/pkg/processor/build/builder.go:267\\nFailed to build processor image\",\"level\":\"warn\",\"message\":\"Failed to create a function; setting the function status\",\"name\":\"deployer\",\"time\":1715845318007.1013}]}}"}
24.05.16 07:41:58.247 (D)     nuctl.platform.docker Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"mkdir -p /etc/nuclio/store/functions/nuclio && /bin/printenv NUCLIO_CONTENTS > /etc/nuclio/store/functions/nuclio/pth-facebookresearch-sam-vit-h.json\"","Stdout":"","Stderr":"","Env":{"NUCLIO_CONTENTS":"eyJtZXRhZGF0YSI6eyJuYW1lIjoicHRoLWZhY2Vib29rcmVzZWFyY2gtc2FtLXZpdC1oIiwibmFtZXNwYWNlIjoibnVjbGlvIiwibGFiZWxzIjp7Im51Y2xpby5pby9wcm9qZWN0LW5hbWUiOiJjdmF0In0sImFubm90YXRpb25zIjp7ImFuaW1hdGVkX2dpZiI6Imh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9vcGVuY3YvY3ZhdC9kZXZlbG9wL3NpdGUvY29udGVudC9lbi9pbWFnZXMvaHJuZXRfZXhhbXBsZS5naWYiLCJmcmFtZXdvcmsiOiJweXRvcmNoIiwiaGVscF9tZXNzYWdlIjoiVGhlIGludGVyYWN0b3IgYWxsb3dzIHRvIGdldCBhIG1hc2sgb2YgYW4gb2JqZWN0IHVzaW5nIGF0IGxlYXN0IG9uZSBwb3NpdGl2ZSwgYW5kIGFueSBuZWdhdGl2ZSBwb2ludHMgaW5zaWRlIGl0IiwibWluX25lZ19wb2ludHMiOiIwIiwibWluX3Bvc19wb2ludHMiOiIxIiwibmFtZSI6IlNlZ21lbnQgQW55dGhpbmciLCJzcGVjIjoiIiwidHlwZSI6ImludGVyYWN0b3IiLCJ2ZXJzaW9uIjoiMiJ9fSwic3BlYyI6eyJkZXNjcmlwdGlvbiI6IkludGVyYWN0aXZlIG9iamVjdCBzZWdtZW50YXRpb24gd2l0aCBTZWdtZW50LUFueXRoaW5nIiwiaGFuZGxlciI6Im1haW46aGFuZGxlciIsInJ1bnRpbWUiOiJweXRob246My44IiwiZW52IjpbeyJuYW1lIjoiUFlUSE9OUEFUSCIsInZhbHVlIjoiL29wdC9udWNsaW8vc2FtIn1dLCJyZXNvdXJjZXMiOnsibGltaXRzIjp7Im52aWRpYS5jb20vZ3B1IjoiMSJ9LCJyZXF1ZXN0cyI6eyJjcHUiOiIyNW0iLCJtZW1vcnkiOiIxTWkifX0sInRhcmdldENQVSI6NzUsInRyaWdnZXJzIjp7Im15SHR0cFRyaWdnZXIiOnsiY2xhc3MiOiIiLCJraW5kIjoiaHR0cCIsIm5hbWUiOiJteUh0dHBUcmlnZ2VyIiwibnVtV29ya2VycyI6MSwid29ya2VyQXZhaWxhYmlsaXR5VGltZW91dE1pbGxpc2Vjb25kcyI6MTAwMDAsImF0dHJpYnV0ZXMiOnsibWF4UmVxdWVzdEJvZHlTaXplIjozMzU1NDQzMn0sIm1heFdvcmtlcnMiOjF9fSwiYnVpbGQiOnsiZnVuY3Rpb25Db25maWdQYXRoIjoicHl0b3JjaC9mYWNlYm9va3Jlc2VhcmNoL3NhbS9udWNsaW8vZnVuY3Rpb24tZ3B1LnlhbWwiLCJpbWFnZSI6ImN2YXQucHRoLmZhY2Vib29rcmVzZWFyY2guc2FtLnZpdF9oIiwiYmFzZUltYWdlIjoidWJ1bnR1OjIyLjA0IiwiZGlyZWN0aXZlcyI6eyJwcmVDb3B5IjpbeyJraW5kIjoiRU5WIiwidmFsdWUiOiJOVklESUFfVklTSUJMRV9ERVZJQ0VTPWFsbCJ9LHsia2luZCI6IkVOViIsInZhbHVlIjoiTlZJRElBX0RSSVZFUl9DQVBBQklMSVRJRVM9Y29tcHV0ZSx1dGlsaXR5In0seyJraW5kIjoiRU5WIiwidmFsdWUiOiJERUJJQU5fRlJPTlRFTkQ9bm9uaW50ZXJhY3RpdmUifSx7ImtpbmQiOiJXT1JLRElSIiwidmFsdWUiOiIvb3B0L251Y2xpby9zYW0ifSx7ImtpbmQiOiJSVU4iLCJ2YWx1ZSI6ImFwdC1nZXQgdXBkYXRlIFx1MDAyNlx1MDAyNiBhcHQtZ2V0IC15IGluc3RhbGwgY3VybCBnaXQgcHl0aG9uMyBweXRob24zLXBpcCBmZm1wZWcgbGlic202IGxpYnhleHQ2In0seyJraW5kIjoiUlVOIiwidmFsdWUiOiJwaXAzIGluc3RhbGwgdG9yY2ggdG9yY2h2aXNpb24gdG9yY2hhdWRpbyBvcGVuY3YtcHl0aG9uIHB5Y29jb3Rvb2xzIG1hdHBsb3RsaWIgb25ueHJ1bnRpbWUgb25ueCJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoicGlwMyBpbnN0YWxsIGdpdCtodHRwczovL2dpdGh1Yi5jb20vZmFjZWJvb2tyZXNlYXJjaC9zZWdtZW50LWFueXRoaW5nLmdpdCJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoiY3VybCAtTyBodHRwczovL2RsLmZiYWlwdWJsaWNmaWxlcy5jb20vc2VnbWVudF9hbnl0aGluZy9zYW1fdml0X2hfNGI4OTM5LnB0aCJ9LHsia2luZCI6IlJVTiIsInZhbHVlIjoibG4gLXMgL3Vzci9iaW4vcGlwMyAvdXNyL2xvY2FsL2Jpbi9waXAgXHUwMDI2XHUwMDI2IGxuIC1zIC91c3IvYmluL3B5dGhvbjMgL3Vzci9iaW4vcHl0aG9uIn1dfSwiY29kZUVudHJ5VHlwZSI6ImltYWdlIn0sInBsYXRmb3JtIjp7ImF0dHJpYnV0ZXMiOnsibW91bnRNb2RlIjoidm9sdW1lIiwicmVzdGFydFBvbGljeSI6eyJtYXhpbXVtUmV0cnlDb3VudCI6MywibmFtZSI6ImFsd2F5cyJ9fX0sInJlYWRpbmVzc1RpbWVvdXRTZWNvbmRzIjoxMjAsInNlY3VyaXR5Q29udGV4dCI6e30sImRpc2FibGVEZWZhdWx0SFRUUFRyaWdnZXIiOmZhbHNlLCJldmVudFRpbWVvdXQiOiIzMHMifSwic3RhdHVzIjp7InN0YXRlIjoiZXJyb3IiLCJtZXNzYWdlIjoiXG5FcnJvciAtIGV4aXQgc3RhdHVzIDFcbiAgICAvbnVjbGlvL3BrZy9jbWRydW5uZXIvc2hlbGxydW5uZXIuZ286MTE0XG5cbkNhbGwgc3RhY2s6XG5zdGRvdXQ6XG5cbnN0ZGVycjpcbkVSUk9SOiBjb3VsZCBub3QgZmluZCAvdG1wL251Y2xpby1idWlsZC0xMDI5NDU3OTkzL3N0YWdpbmc6IHN0YXQgL3RtcC9udWNsaW8tYnVpbGQtMTAyOTQ1Nzk5My9zdGFnaW5nOiBubyBzdWNoIGZpbGUgb3IgZGlyZWN0b3J5XG5cbiAgICAvbnVjbGlvL3BrZy9jbWRydW5uZXIvc2hlbGxydW5uZXIuZ286MTE0XG5GYWlsZWQgdG8gYnVpbGRcbiAgICAvbnVjbGlvL3BrZy9kb2NrZXJjbGllbnQvc2hlbGwuZ286MTE5XG5GYWlsZWQgdG8gYnVpbGQgb25idWlsZCBpbWFnZVxuICAgIC4uLi9wa2cvY29udGFpbmVyaW1hZ2VidWlsZGVycHVzaGVyL2RvY2tlci5nbzoyNTdcbkZhaWxlZCB0byBjb3B5IG9iamVjdHMgZnJvbSBvbmJ1aWxkXG4gICAgLi4uL3BrZy9jb250YWluZXJpbWFnZWJ1aWxkZXJwdXNoZXIvZG9ja2VyLmdvOjIxN1xuRmFpbGVkIHRvIGJ1aWxkIGltYWdlIGFydGlmYWN0c1xuICAgIC4uLi9wa2cvY29udGFpbmVyaW1hZ2VidWlsZGVycHVzaGVyL2RvY2tlci5nbzo2NlxuRmFpbGVkIHRvIGJ1aWxkIHByb2Nlc3NvciBpbWFnZVxuICAgIC9udWNsaW8vcGtnL3Byb2Nlc3Nvci9idWlsZC9idWlsZGVyLmdvOjI2N1xuIiwibG9ncyI6W3sibGV2ZWwiOiJpbmZvIiwibWVzc2FnZSI6IkRlcGxveWluZyBmdW5jdGlvbiIsIm5hbWUiOiJwdGgtZmFjZWJvb2tyZXNlYXJjaC1zYW0tdml0LWgiLCJ0aW1lIjoxNzE1ODQ1MzE2MzIwLjIzOH0seyJidWlsZGVyS2luZCI6ImRvY2tlciIsImxldmVsIjoiaW5mbyIsIm1lc3NhZ2UiOiJCdWlsZGluZyIsIm5hbWUiOiJwdGgtZmFjZWJvb2tyZXNlYXJjaC1zYW0tdml0LWgiLCJ0aW1lIjoxNzE1ODQ1MzE2MzIwLjI4MjcsInZlcnNpb25JbmZvIjoiTGFiZWw6IDEuMTMuMCwgR2l0IGNvbW1pdDogYzQ0MjJlYjc3Mjc4MWZiNTBmYmYwMTc2OThhYWU5NjE5OWQ4MTM4OCwgT1M6IGxpbnV4LCBBcmNoOiBhbWQ2NCwgR28gdmVyc2lvbjogZ28xLjIxLjcifSx7ImxldmVsIjoiaW5mbyIsIm1lc3NhZ2UiOiJTdGFnaW5nIGZpbGVzIGFuZCBwcmVwYXJpbmcgYmFzZSBpbWFnZXMiLCJuYW1lIjoiZGVwbG95ZXIiLCJ0aW1lIjoxNzE1ODQ1MzE2NjM0LjQwMzN9LHsibGV2ZWwiOiJ3YXJuIiwibWVzc2FnZSI6IlB5dGhvbiAzLjggcnVudGltZSBpcyBkZXByZWNhdGVkIGFuZCB3aWxsIHNvb24gbm90IGJlIHN1cHBvcnRlZC4gTWlncmF0ZSB5b3VyIGNvZGUgYW5kIHVzZSBQeXRob24gMy45IHJ1bnRpbWUgKGBweXRob246My45YCkgb3IgaGlnaGVyIiwibmFtZSI6ImRlcGxveWVyIiwidGltZSI6MTcxNTg0NTMxNjYzNC44MDY2fSx7ImJhc2VJbWFnZSI6InVidW50dToyMi4wNCIsImxldmVsIjoid2FybiIsIm1lc3NhZ2UiOiJVc2luZyB1c2VyIHByb3ZpZGVkIGJhc2UgaW1hZ2UsIHJ1bnRpbWUgaW50ZXJwcmV0ZXIgdmVyc2lvbiBpcyBwcm92aWRlZCBieSB0aGUgYmFzZSBpbWFnZSIsIm5hbWUiOiJkZXBsb3llciIsInRpbWUiOjE3MTU4NDUzMTY2MzQuODI4fSx7ImxldmVsIjoiaW5mbyIsIm1lc3NhZ2UiOiJCdWlsZGluZyBwcm9jZXNzb3IgaW1hZ2UiLCJuYW1lIjoiZGVwbG95ZXIiLCJyZWdpc3RyeVVSTCI6IiIsInRhZ2dlZEltYWdlTmFtZSI6ImN2YXQucHRoLmZhY2Vib29rcmVzZWFyY2guc2FtLnZpdF9oOmxhdGVzdCIsInRpbWUiOjE3MTU4NDUzMTY2MzUuMDU2Mn0seyJlcnIiOiJGYWlsZWQgdG8gYnVpbGQgcHJvY2Vzc29yIGltYWdlIiwiZXJyVmVyYm9zZSI6IlxuRXJyb3IgLSBleGl0IHN0YXR1cyAxXG4gICAgL251Y2xpby9wa2cvY21kcnVubmVyL3NoZWxscnVubmVyLmdvOjExNFxuXG5DYWxsIHN0YWNrOlxuc3Rkb3V0OlxuXG5zdGRlcnI6XG5FUlJPUjogY291bGQgbm90IGZpbmQgL3RtcC9udWNsaW8tYnVpbGQtMTAyOTQ1Nzk5My9zdGFnaW5nOiBzdGF0IC90bXAvbnVjbGlvLWJ1aWxkLTEwMjk0NTc5OTMvc3RhZ2luZzogbm8gc3VjaCBmaWxlIG9yIGRpcmVjdG9yeVxuXG4gICAgL251Y2xpby9wa2cvY21kcnVubmVyL3NoZWxscnVubmVyLmdvOjExNFxuRmFpbGVkIHRvIGJ1aWxkXG4gICAgL251Y2xpby9wa2cvZG9ja2VyY2xpZW50L3NoZWxsLmdvOjExOVxuRmFpbGVkIHRvIGJ1aWxkIG9uYnVpbGQgaW1hZ2VcbiAgICAuLi4vcGtnL2NvbnRhaW5lcmltYWdlYnVpbGRlcnB1c2hlci9kb2NrZXIuZ286MjU3XG5GYWlsZWQgdG8gY29weSBvYmplY3RzIGZyb20gb25idWlsZFxuICAgIC4uLi9wa2cvY29udGFpbmVyaW1hZ2VidWlsZGVycHVzaGVyL2RvY2tlci5nbzoyMTdcbkZhaWxlZCB0byBidWlsZCBpbWFnZSBhcnRpZmFjdHNcbiAgICAuLi4vcGtnL2NvbnRhaW5lcmltYWdlYnVpbGRlcnB1c2hlci9kb2NrZXIuZ286NjZcbkZhaWxlZCB0byBidWlsZCBwcm9jZXNzb3IgaW1hZ2VcbiAgICAvbnVjbGlvL3BrZy9wcm9jZXNzb3IvYnVpbGQvYnVpbGRlci5nbzoyNjdcbkZhaWxlZCB0byBidWlsZCBwcm9jZXNzb3IgaW1hZ2UiLCJsZXZlbCI6Indhcm4iLCJtZXNzYWdlIjoiRmFpbGVkIHRvIGNyZWF0ZSBhIGZ1bmN0aW9uOyBzZXR0aW5nIHRoZSBmdW5jdGlvbiBzdGF0dXMiLCJuYW1lIjoiZGVwbG95ZXIiLCJ0aW1lIjoxNzE1ODQ1MzE4MDA3LjEwMTN9XX19"}}}

Error - exit status 1
    /nuclio/pkg/cmdrunner/shellrunner.go:114

Call stack:
stdout:

stderr:
ERROR: could not find /tmp/nuclio-build-1029457993/staging: stat /tmp/nuclio-build-1029457993/staging: no such file or directory

    /nuclio/pkg/cmdrunner/shellrunner.go:114
Failed to build
    /nuclio/pkg/dockerclient/shell.go:119
Failed to build onbuild image
    .../pkg/containerimagebuilderpusher/docker.go:257
Failed to copy objects from onbuild
    .../pkg/containerimagebuilderpusher/docker.go:217
Failed to build image artifacts
    .../pkg/containerimagebuilderpusher/docker.go:66

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rokatyy @jaroslavknotek