Custom loggging of prompts and completion in the case of AI Services

[PrimosK]

Hi everyone,

I have a requirement to log each prompt (sequence of messages) and completion (response) including statistics (prompt tokens, total tokens, completion tokens) into a dedicated log/textual file in a custom format.

Now this can be easily achieved using ChatLanguageModel API:

  Response<AiMessage> generate(List<ChatMessage> messages);

... as I can log all necessary information by wrapping this call and extracting data from Response, AiMessage, and List<ChatMessage>.

Is there any way to do the same when using AI Services API.

I understand that I can use structured output like Response<AiMessage> which would give me the message and statistics... But still - I would not have infomration of what text (messages) was actually passed to the model.

[langchain4j]

Hi @PrimosK we are working on this right now, could you please take a look and provide a feedback? Here is a very early draft: #1058

[PrimosK]

It's really nice to see this was already took into consideration.

I am thinking whether it would make sense:

• for response listener to look like: default void onResponse(ChatLanguageModelRequest request, ChatLanguageModelResponse response) ... which would give you instant access to the request that was used to produce the response so there would be no need to correlate response with particular request using id manually.
• add ˙onError(ChatLanguageModelRequest request, Exception e)˙ listener which would be called in catch block surrounding chatCompletition related code

[langchain4j]

Hi @PrimosK thank you, that totally makes sense!