TruLens/Pydantic Compatibility

[CamDuffy1]

TruLens is a tool that can be used to objectively evaluate a RAG application based on the following metrics (among others):

1. Answer relevance (How relevant is the response to the query?)
2. Context relevance (How relevant is the retrieved context to the query?)
3. Groundedness (How well does the retrieved context support the response?)

trulens-eval is the Python library used to implement TruLens through code. Unfortunately, there are conflicting package dependencies between trulens-eval and other libraries used within SEC Insights. This prevents the use of TruLens to evaluate the RAG performance of SEC Insights.

The latest version of trulens-eval (0.23.0 at the time of writing) conflicts with the version of Pydantic required by the SEC Insights backend.

0.17.0 is the latest version of trulens-eval that is compatible with the required version of Pydantic. However, this version of trulens-eval needs an older version of the OpenAI library, which conflicts with the version needed by llama-index. Because of this, trulens-eval 0.17.0 can't be used to create the feedback functions it needs to evaluate RAG performance.

Pydantic version >=2.0 is required by more recent versions of trulens-eval (such as 0.23.0) that are compatible with the OpenAI version (1.5.0) used by SEC Insights. The Pydantic Schemas for the API component of SEC Insights were written using Pydantic version ^1.10.8. Pydantic v2 introduces breaking changes from earlier versions, so legacy Pydantic code needs to be updated for compatibility with versions >=2.0.

Pydantic v2 still includes its v1 features, which can be accessed by importing through pydantic.v1. I installed Pydantic 2.6.1 and updated the existing Pydantic code in SEC Insights to be imported this way. This resulted in an internal server error (500) when trying to access the API.

(from the browser at https://<codespace-name>-8000.app.github.dev/docs)

(from the command line)

Pydantic provides a code transformation tool to help with migrating code from v1 to v2. I used this tool to update the Pydantic code in the backend directory, however I was not able to start the backend after doing so. Upon running the make run command from the backend directory, I received the following error related to Pydantic models:

Is there a known workaround to make SEC Insights compatible with Pydantic v2, or plans to upgrade the backend Pydantic code?

[ashleych]

Facing similar issues with pydantic