ops: fractional GPU usage for embedding server within kubernetes

[cdxker]

Description

Currently the embedding server is not ran in Kubernetes, this is an issue since we can't scale the embedding servers easily.

Community channels

Matrix is preferred. Reach out on discord or Matrix for further assistance.

• discord
• matrix

[cdxker]

Did find this example for fractional GPU usage here. https://github.com/DanielKneipp/aws-eks-share-gpu

[cdxker]

Above link is out of date and needs to be updated. Looking into this to solve the issue
https://github.com/NVIDIA/k8s-device-plugin?tab=readme-ov-file#shared-access-to-gpus

[cdxker]

Gave up on attempting this in AWS, it worked instantly with google cloud platform