Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Image Bounding Box Annotations for vision & multimodal workflows #4739

Open
JonnyTran opened this issue Apr 23, 2024 · 0 comments
Open
Labels
area: ui Indicates that an issue or pull request is related to the User Interface (UI) team: ml Indicates that the issue or pull request is owned by the Machine Learning (ML) team type: enhancement Indicates new feature requests

Comments

@JonnyTran
Copy link

JonnyTran commented Apr 23, 2024

Hi team!
Given the awesome SpanQuestion new feature recently released, I'm tempted to ask if it's possible to have the same done for annotating regions of interests for images. It would be marvelous if there's a way to draw and label rectangle bounding boxes on an ImageField, similar to LabelStudio's bounding boxes. I love Argilla and hope this feature can be on the roadmap!

Is your feature request related to a problem? Please describe.
As LLMs are increasingly becoming multimodal and many data workflows involve a mixture of text, document, and image data types, a very common task is to highlight specific regions in an image for annotation or other downstream processing. For a specific use case, suppose an LLM is tasked with detecting objects and return a bounding box of the detected objects in JSON. It would be great to be able to take this JSON output and have humans add, edit or label the bboxes, which can either be used for finetuning multimodal LLMs or downstream tasks.

Describe the solution you'd like
Currently images are displayed using image_to_html to encode images as html in a TextField. We can create a ImageField to contain an image url and other metadata (width, height, scale, offsets), and a ImageSpanQuestion where users can create RectangleLabels and PointLabels stored as responses or suggestions.

Describe alternatives you've considered
There isn't really a workaround to annotating region of interests without drawing on top of the image.

@dosubot dosubot bot added area: ui Indicates that an issue or pull request is related to the User Interface (UI) team: ml Indicates that the issue or pull request is owned by the Machine Learning (ML) team type: enhancement Indicates new feature requests labels Apr 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: ui Indicates that an issue or pull request is related to the User Interface (UI) team: ml Indicates that the issue or pull request is owned by the Machine Learning (ML) team type: enhancement Indicates new feature requests
Projects
None yet
Development

No branches or pull requests

1 participant