[Bug]: intro_multimodal_rag: Vertex AI API Call Results in _MultiThreadedRendezvous/InvalidArgument Error

[ab-kotecha]

Contact Details

abhishek@datavtar.com

File Name

gemini/getting-started/intro_gemini_python.ipynb

What happened?

Summary

Encounter an InvalidArgument error when executing a content generation request using Vertex AI's API in a custom processing workflow for PDF documents. The error occurs within the get_gemini_response function, disrupting the extraction and processing of image and text metadata.

Steps to Reproduce

1. Initialize a workflow to process PDF documents using Vertex AI Workbench, extracting image and text metadata.
2. Execute the get_document_metadata function with a valid PDF document, specifying parameters for image description generation.
3. Observe an InvalidArgument error during the execution of get_gemini_response, specifically when calling generative_multimodal_model.generate_content.

Expected Behavior

The expected behavior is successful generation of content descriptions for images extracted from PDF documents without encountering an InvalidArgument error.

Actual Behavior

The process fails, triggering an _MultiThreadedRendezvous that leads to an InvalidArgument error. The traceback indicates an issue with the content generation request to Vertex AI's API.

Environment

• Execution environment: Vertex AI Workbench
• Python version: 3.11
• Vertex AI Workbench Environment: Python 3 (with Intel® MKL)
• Vertex AI Workbench Environment version: M117

Additional Context

• This issue persists despite various attempts to validate and format the input correctly according to the API's expected parameters.

Possible Causes and Solutions

• A potential cause could be a mismatch or incorrect formatting of the input parameters to the generate_content API call.
• Verifying the compatibility and formatting of all input arguments against the latest API documentation might provide insight.

Relevant log output

---------------------------------------------------------------------------
_MultiThreadedRendezvous                  Traceback (most recent call last)
File /opt/conda/lib/python3.10/site-packages/google/api_core/grpc_helpers.py:155, in _wrap_stream_errors.<locals>.error_remapped_callable(*args, **kwargs)
    154     prefetch_first = getattr(callable_, "_prefetch_first_result_", True)
--> 155     return _StreamingResponseIterator(
    156         result, prefetch_first_result=prefetch_first
    157     )
    158 except grpc.RpcError as exc:

File /opt/conda/lib/python3.10/site-packages/google/api_core/grpc_helpers.py:81, in _StreamingResponseIterator.__init__(self, wrapped, prefetch_first_result)
     80     if prefetch_first_result:
---> 81         self._stored_first_result = next(self._wrapped)
     82 except TypeError:
     83     # It is possible the wrapped method isn't an iterable (a grpc.Call
     84     # for instance). If this happens don't store the first result.

File /opt/conda/lib/python3.10/site-packages/grpc/_channel.py:540, in _Rendezvous.__next__(self)
    539 def __next__(self):
--> 540     return self._next()

File /opt/conda/lib/python3.10/site-packages/grpc/_channel.py:966, in _MultiThreadedRendezvous._next(self)
    965 elif self._state.code is not None:
--> 966     raise self

_MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "Request contains an invalid argument."
	debug_error_string = "UNKNOWN:Error received from peer ipv4:108.177.112.95:443 {created_time:"2024-02-28T05:37:43.16371168+00:00", grpc_status:3, grpc_message:"Request contains an invalid argument."}"
>

The above exception was the direct cause of the following exception:

InvalidArgument                           Traceback (most recent call last)
Cell In[18], line 14
      7 image_description_prompt = """Explain what is going on in the image.
      8 If it's a table, extract all elements of the table.
      9 If it's a graph, explain the findings in the graph.
     10 Do not include any numbers that are not mentioned in the image.
     11 """
     13 # Extract text and image metadata from the PDF document
---> 14 text_metadata_df, image_metadata_df = get_document_metadata(
     15     multimodal_model,
     16     pdf_folder_path,
     17     image_save_dir="./images",
     18     image_description_prompt=image_description_prompt,
     19     embedding_size=1408,
     20     # add_sleep_after_page = True, # Uncomment this if you are running into API quota issues
     21     sleep_time_after_page = 5,
     22     generation_config = GenerationConfig(temperature=0.2, max_output_tokens=2048),# see next cell
     23     safety_settings = {
     24         HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_NONE,
     25         HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_NONE,
     26         HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: HarmBlockThreshold.BLOCK_NONE,
     27         HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE,
     28     }, # see next cell
     29 )
     31 print("\n\n --- Completed processing. ---")

File ~/utils/intro_multimodal_rag_utils.py:545, in get_document_metadata(generative_multimodal_model, pdf_folder_path, image_save_dir, image_description_prompt, embedding_size, generation_config, safety_settings, add_sleep_after_page, sleep_time_after_page)
    537 image_for_gemini, image_name = get_image_for_gemini(
    538     doc, image, image_no, image_save_dir, file_name, page_num
    539 )
    541 print(
    542     f"Extracting image from page: {page_num + 1}, saved as: {image_name}"
    543 )
--> 545 response = get_gemini_response(
    546     generative_multimodal_model,
    547     model_input=[image_description_prompt, image_for_gemini],
    548     generation_config=generation_config,
    549     safety_settings=safety_settings,
    550     stream=True,
    551 )
    553 image_embedding = get_image_embedding_from_multimodal_embedding_model(
    554     image_uri=image_name,
    555     embedding_size=embedding_size,
    556 )
    558 image_description_text_embedding = (
    559     get_text_embedding_from_text_embedding_model(text=response)
    560 )

File ~/utils/intro_multimodal_rag_utils.py:363, in get_gemini_response(generative_multimodal_model, model_input, stream, generation_config, safety_settings)
    355 response = generative_multimodal_model.generate_content(
    356     model_input,
    357     generation_config=generation_config,
    358     stream=stream,
    359     safety_settings=safety_settings,
    360 )
    361 response_list = []
--> 363 for chunk in response:
    364     try:
    365         response_list.append(chunk.text)

File ~/.local/lib/python3.10/site-packages/vertexai/generative_models/_generative_models.py:505, in _GenerativeModel._generate_content_streaming(self, contents, generation_config, safety_settings, tools)
    482 """Generates content.
    483 
    484 Args:
   (...)
    497     A stream of GenerationResponse objects
    498 """
    499 request = self._prepare_request(
    500     contents=contents,
    501     generation_config=generation_config,
    502     safety_settings=safety_settings,
    503     tools=tools,
    504 )
--> 505 response_stream = self._prediction_client.stream_generate_content(
    506     request=request
    507 )
    508 for chunk in response_stream:
    509     yield self._parse_response(chunk)

File ~/.local/lib/python3.10/site-packages/google/cloud/aiplatform_v1beta1/services/prediction_service/client.py:2202, in PredictionServiceClient.stream_generate_content(self, request, model, contents, retry, timeout, metadata)
   2199 self._validate_universe_domain()
   2201 # Send the request.
-> 2202 response = rpc(
   2203     request,
   2204     retry=retry,
   2205     timeout=timeout,
   2206     metadata=metadata,
   2207 )
   2209 # Done; return the response.
   2210 return response

File /opt/conda/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py:113, in _GapicCallable.__call__(self, timeout, retry, *args, **kwargs)
    110     metadata.extend(self._metadata)
    111     kwargs["metadata"] = metadata
--> 113 return wrapped_func(*args, **kwargs)

File /opt/conda/lib/python3.10/site-packages/google/api_core/grpc_helpers.py:159, in _wrap_stream_errors.<locals>.error_remapped_callable(*args, **kwargs)
    155     return _StreamingResponseIterator(
    156         result, prefetch_first_result=prefetch_first
    157     )
    158 except grpc.RpcError as exc:
--> 159     raise exceptions.from_grpc_error(exc) from exc

InvalidArgument: 400 Request contains an invalid argument.

Code of Conduct

• I agree to follow this project's Code of Conduct

[ab-kotecha]

Added Python version details.

[ab-kotecha]

Updated environment details for VertexAI Workbench.

[lavinigam-gcp]

Hi @ab-kotecha,

Thank you for raising the issue.

As a quick fix, can you please enable “add_sleep_after_page" as True and then re-run this block:

text_metadata_df, image_metadata_df = get_document_metadata(
    multimodal_model,  # we are passing gemini 1.0 pro vision model
    pdf_folder_path,
    image_save_dir="images",
    image_description_prompt=image_description_prompt,
    embedding_size=1408,
    add_sleep_after_page = True, # Uncomment this if you are running into API quota issues
    sleep_time_after_page = 5,
    # generation_config = # see next cell
    # safety_settings =  # see next cell
)

If you can also share few more details about your workflow, it would really help me zero down the issues and I can help and support you better:

• How many documents are you passing and how many pages does that have overall?
• Does your documents contain a mix of images and text?
• Are you getting '_MultiThreadedRendezvous' error right at the start of the processing, or it happens after few pages?

[ab-kotecha]

Thanks for your suggestion Lavi. I have kept the settings for add_sleep_after_page and sleep_time_after_page by uncommenting them earlier for both Vertex AI Workbench and Google Colab platform. With and without those settings active for all.

How many documents are you passing and how many pages does that have overall?
-> I am using the same documents as in the demo, the Alphabet 10K report, no change. Running the script sequentially without a single change.
Does your documents contain a mix of images and text?
-> Yes. The document is the same which is part of the demo story.
Are you getting '_MultiThreadedRendezvous' error right at the start of the processing, or it happens after few pages?
-> It appears as the first image goes for extract processing.
Processing the file: --------------------------------- data/google-10k-sample-part1.pdf

Processing page: 1
Processing page: 2
Extracting image from page: 2, saved as: images/google-10k-sample-part1.pdf_image_1_0_11.jpeg
ERROR Occurs as soon as the above line is printed on Jupyter.

---

Strangely enough, I did the same thing in the Skill Boost environment lab, that issue was not there. Then I copied the notebook content from lab and pasted the file in my personal account, the issue appeared again. I was able to reproduce the same error on Vertex AI Workbench and Google Colab, with or without the add_sleep_after_page and sleep_time_after_page configs.

[ab-kotecha]

I tried again, the following is the cell output for the code below:

# Specify the PDF folder with multiple PDF

# pdf_folder_path = "/content/data/" # if running in Google Colab/Colab Enterprise
pdf_folder_path = "data/"  # if running in Vertex AI Workbench.

# Specify the image description prompt. Change it
image_description_prompt = """Explain what is going on in the image.
If it's a table, extract all elements of the table.
If it's a graph, explain the findings in the graph.
Do not include any numbers that are not mentioned in the image.
"""

# Extract text and image metadata from the PDF document
text_metadata_df, image_metadata_df = get_document_metadata(
    multimodal_model,  # we are passing gemini 1.0 pro vision model
    pdf_folder_path,
    image_save_dir="images",
    image_description_prompt=image_description_prompt,
    embedding_size=1408,
    add_sleep_after_page = True, # Uncomment this if you are running into API quota issues
    sleep_time_after_page = 5,
    # generation_config = # see next cell
    # safety_settings =  # see next cell
)

print("\n\n --- Completed processing. ---")

---

 Processing the file: --------------------------------- data/google-10k-sample-part2.pdf 


Processing page: 1
Extracting image from page: 1, saved as: images/google-10k-sample-part2.pdf_image_0_0_6.jpeg
{
	"name": "InvalidArgument",
	"message": "400 Request contains an invalid argument.",
	"stack": "---------------------------------------------------------------------------
_MultiThreadedRendezvous                  Traceback (most recent call last)
File ~/stage5-ip/.venv_stage5/lib/python3.11/site-packages/google/api_core/grpc_helpers.py:173, in _wrap_stream_errors.<locals>.error_remapped_callable(*args, **kwargs)
    172     prefetch_first = getattr(callable_, \"_prefetch_first_result_\", True)
--> 173     return _StreamingResponseIterator(
    174         result, prefetch_first_result=prefetch_first
    175     )
    176 except grpc.RpcError as exc:

File ~/stage5-ip/.venv_stage5/lib/python3.11/site-packages/google/api_core/grpc_helpers.py:95, in _StreamingResponseIterator.__init__(self, wrapped, prefetch_first_result)
     94     if prefetch_first_result:
---> 95         self._stored_first_result = next(self._wrapped)
     96 except TypeError:
     97     # It is possible the wrapped method isn't an iterable (a grpc.Call
     98     # for instance). If this happens don't store the first result.

File ~/stage5-ip/.venv_stage5/lib/python3.11/site-packages/grpc/_channel.py:540, in _Rendezvous.__next__(self)
    539 def __next__(self):
--> 540     return self._next()

File ~/stage5-ip/.venv_stage5/lib/python3.11/site-packages/grpc/_channel.py:966, in _MultiThreadedRendezvous._next(self)
    965 elif self._state.code is not None:
--> 966     raise self

_MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
\tstatus = StatusCode.INVALID_ARGUMENT
\tdetails = \"Request contains an invalid argument.\"
\tdebug_error_string = \"UNKNOWN:Error received from peer ipv6:%5B2404:6800:4009:815::200a%5D:443 {created_time:\"2024-03-01T11:58:50.247534+05:30\", grpc_status:3, grpc_message:\"Request contains an invalid argument.\"}\"
>

The above exception was the direct cause of the following exception:

InvalidArgument                           Traceback (most recent call last)
Cell In[9], line 14
      7 image_description_prompt = \"\"\"Explain what is going on in the image.
      8 If it's a table, extract all elements of the table.
      9 If it's a graph, explain the findings in the graph.
     10 Do not include any numbers that are not mentioned in the image.
     11 \"\"\"
     13 # Extract text and image metadata from the PDF document
---> 14 text_metadata_df, image_metadata_df = get_document_metadata(
     15     multimodal_model,  # we are passing gemini 1.0 pro vision model
     16     pdf_folder_path,
     17     image_save_dir=\"images\",
     18     image_description_prompt=image_description_prompt,
     19     embedding_size=1408,
     20     add_sleep_after_page = True, # Uncomment this if you are running into API quota issues
     21     sleep_time_after_page = 5,
     22     # generation_config = # see next cell
     23     # safety_settings =  # see next cell
     24 )
     26 print(\"\
\
 --- Completed processing. ---\")

File ~/stage5-ip/playground/read/utils/intro_multimodal_rag_utils.py:545, in get_document_metadata(generative_multimodal_model, pdf_folder_path, image_save_dir, image_description_prompt, embedding_size, generation_config, safety_settings, add_sleep_after_page, sleep_time_after_page)
    537 image_for_gemini, image_name = get_image_for_gemini(
    538     doc, image, image_no, image_save_dir, file_name, page_num
    539 )
    541 print(
    542     f\"Extracting image from page: {page_num + 1}, saved as: {image_name}\"
    543 )
--> 545 response = get_gemini_response(
    546     generative_multimodal_model,
    547     model_input=[image_description_prompt, image_for_gemini],
    548     generation_config=generation_config,
    549     safety_settings=safety_settings,
    550     stream=True,
    551 )
    553 image_embedding = get_image_embedding_from_multimodal_embedding_model(
    554     image_uri=image_name,
    555     embedding_size=embedding_size,
    556 )
    558 image_description_text_embedding = (
    559     get_text_embedding_from_text_embedding_model(text=response)
    560 )

File ~/stage5-ip/playground/read/utils/intro_multimodal_rag_utils.py:363, in get_gemini_response(generative_multimodal_model, model_input, stream, generation_config, safety_settings)
    355 response = generative_multimodal_model.generate_content(
    356     model_input,
    357     generation_config=generation_config,
    358     stream=stream,
    359     safety_settings=safety_settings,
    360 )
    361 response_list = []
--> 363 for chunk in response:
    364     try:
    365         response_list.append(chunk.text)

File ~/stage5-ip/.venv_stage5/lib/python3.11/site-packages/vertexai/generative_models/_generative_models.py:505, in _GenerativeModel._generate_content_streaming(self, contents, generation_config, safety_settings, tools)
    482 \"\"\"Generates content.
    483 
    484 Args:
   (...)
    497     A stream of GenerationResponse objects
    498 \"\"\"
    499 request = self._prepare_request(
    500     contents=contents,
    501     generation_config=generation_config,
    502     safety_settings=safety_settings,
    503     tools=tools,
    504 )
--> 505 response_stream = self._prediction_client.stream_generate_content(
    506     request=request
    507 )
    508 for chunk in response_stream:
    509     yield self._parse_response(chunk)

File ~/stage5-ip/.venv_stage5/lib/python3.11/site-packages/google/cloud/aiplatform_v1beta1/services/prediction_service/client.py:2207, in PredictionServiceClient.stream_generate_content(self, request, model, contents, retry, timeout, metadata)
   2204 self._validate_universe_domain()
   2206 # Send the request.
-> 2207 response = rpc(
   2208     request,
   2209     retry=retry,
   2210     timeout=timeout,
   2211     metadata=metadata,
   2212 )
   2214 # Done; return the response.
   2215 return response

File ~/stage5-ip/.venv_stage5/lib/python3.11/site-packages/google/api_core/gapic_v1/method.py:131, in _GapicCallable.__call__(self, timeout, retry, compression, *args, **kwargs)
    128 if self._compression is not None:
    129     kwargs[\"compression\"] = compression
--> 131 return wrapped_func(*args, **kwargs)

File ~/stage5-ip/.venv_stage5/lib/python3.11/site-packages/google/api_core/grpc_helpers.py:177, in _wrap_stream_errors.<locals>.error_remapped_callable(*args, **kwargs)
    173     return _StreamingResponseIterator(
    174         result, prefetch_first_result=prefetch_first
    175     )
    176 except grpc.RpcError as exc:
--> 177     raise exceptions.from_grpc_error(exc) from exc

InvalidArgument: 400 Request contains an invalid argument."
}

[ab-kotecha]

I tried this again today in different environments. I am getting the same issue. Not sure if I need to perform any operation to the GCloud API/Limits?

[lavinigam-gcp]

HI @ab-kotecha, It seems like a localized issue at your end, possibly something to do with your Access or Quota. I tested the notebook again with my personal GCP account and the notebook seems to be working fine. Are you using personal GCP account (and on free $300 credits) or corporate account?

[ab-kotecha]

Hi Lavi, Thanks for your email. I tested this on a corporate account. Which quote/limit do you think I should use? I checked all the quota on the Service Limits, and none of them are being hit. I am not sure if there is any additional API that I need to enable? Best, Abhishek

…

On Fri, 8 Mar 2024 at 11:21, Lavi Nigam ***@***.***> wrote: HI @ab-kotecha <https://github.com/ab-kotecha>, It seems like a localized issue at your end, possibly something to do with your Access or Quota. I tested the notebook again with my personal GCP account and the notebook seems to be working fine. Are you using personal GCP account (and on free $300 credits) or corporate account? — Reply to this email directly, view it on GitHub <#427 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APQIJDKXEJWTJJMO4TRX6QDYXFGVLAVCNFSM6AAAAABD5LX2HKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBVGA3TSNBRGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>