Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Gemini evaluation: string format error and missing class definition #652

Open
1 task done
pepecura opened this issue May 3, 2024 · 0 comments
Open
1 task done
Assignees

Comments

@pepecura
Copy link

pepecura commented May 3, 2024

File Name

https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/get_started_with_genai_model_eval_service.ipynb

What happened?

  1. The string prompt_end gives an error in "Evaluate prompt engineering using locally-defined custom metrics" section. The fix: Add f at the beginning of first prompt_end and delete the fourth quotation mark at the end of the second prompt_end.
  2. Cosmetics: Delete from BUCKET_URI definition.
  3. Metric list gives an error in "Validate prompt by comparing LLM 1 with LLM 2". The fix: Delete rouge_1 and rouge_2 from metrics.
  4. class EvaluationTextSummarizationSpec in "Using Vertex AI Model Evaluation Computation-based metrics" is not imported from language_models and gives an error. It doesn't exist in this version of the module (it is in preview).
    Thanks!

Relevant log output

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants