[BUG] java.lang.NullPointerException - "tokenUsageAccumulator" is null - when using LocalAI function calling

[rprabhu]

Describe the bug

When implementing function calling using LocalAI a NPE is thrown stating "tokenUsageAccumulator" is null

Log and Stack trace

Exception in thread "main" java.lang.NullPointerException: Cannot invoke "dev.langchain4j.model.output.TokenUsage.add(dev.langchain4j.model.output.TokenUsage)" because "tokenUsageAccumulator" is null
        at dev.langchain4j.service.DefaultAiServices$1.invoke(DefaultAiServices.java:175)
        at $Proxy3.chat(Unknown Source)
        at AiServicesFunctionCalling.main(AiServicesFunctionCalling.java:71)

To Reproduce

Following is the jbang script

//DEPS dev.langchain4j:langchain4j:0.29.1
//DEPS dev.langchain4j:langchain4j-local-ai:0.29.1


import java.time.Duration;
import dev.langchain4j.agent.tool.Tool;
import dev.langchain4j.memory.ChatMemory;
import dev.langchain4j.memory.chat.MessageWindowChatMemory;
import dev.langchain4j.model.chat.ChatLanguageModel;
import dev.langchain4j.model.localai.LocalAiChatModel;
import dev.langchain4j.service.AiServices;

class AiServicesFunctionCalling {

    private static final String MODEL = "gpt-4";
    private static final String BASE_URL = "http://localhost:8080";
    private static final Duration timeout = Duration.ofSeconds(120);
    private static String question;

    static class Calculator {

        @Tool("Calculates the length of a string")
        int stringLength(String s) {
            System.out.println("Called stringLength with s='" + s + "'");
            return s.length();
        }

        @Tool("Calculates the sum of two numbers")
        int add(int a, int b) {
            System.out.println("Called add with a=" + a + ", b=" + b);
            return a + b;
        }

        @Tool("Calculates the square root of a number")
        double sqrt(int x) {
            System.out.println("Called sqrt with x=" + x);
            return Math.sqrt(x);
        }
    }

    interface ChatMinion {
        String chat(String message);
    }

    public static void main(String[] args) {

        ChatLanguageModel model = LocalAiChatModel.builder()
                .baseUrl(BASE_URL)
                .modelName(MODEL)
                .timeout(timeout)
                .temperature(0.0)
                .build();
        ChatMemory memory = MessageWindowChatMemory.withMaxMessages(10);

        ChatMinion minion = AiServices.builder(ChatMinion.class)
                .chatLanguageModel(model)
                .tools(new Calculator())
                .chatMemory(memory)
                .build();
        question = "What is the square root 226?";
        System.out.println("Asking question: " + question);
        String text = minion.chat(question);
        System.out.println("\n\nResponse:\n\n" + text);
    }
}

Expected behavior

Should call the function and return the result

Please complete the following information:

• LangChain4j version: 0.29.1
• LLM(s) used: LocalAI (gpt-4)
• Java version: 21
• Spring Boot version (if applicable): NA

Additional context

[Kugaaa]

dev.langchain4j.model.localai.LocalAiChatModel#generate(java.util.List<dev.langchain4j.data.message.ChatMessage>, java.util.List<dev.langchain4j.agent.tool.ToolSpecification>, dev.langchain4j.agent.tool.ToolSpecification)

private Response<AiMessage> generate(List<ChatMessage> messages,
                                         List<ToolSpecification> toolSpecifications,
                                         ToolSpecification toolThatMustBeExecuted
    ) {
        ChatCompletionRequest.Builder requestBuilder = ChatCompletionRequest.builder()
                .model(modelName)
                .messages(toOpenAiMessages(messages))
                .temperature(temperature)
                .topP(topP)
                .maxTokens(maxTokens);
        ...
        return Response.from(
                aiMessageFrom(response),
                null,
                finishReasonFrom(response.choices().get(0).finishReason())
        );
    }

the response of generate returned that tokenUsage is null
So, is null here the correct situation? I have considered two ways

• proxy method invoke compatible null value
• diffrent TokenUsage implement

I hope to try to solve it~

[langchain4j]

Related to #921

[langchain4j]

@Kugaaa I think AI Service should be changed to handle TokenUsage == null situation well, not all LLM providers return it.

Ideally, LocalAiChatModel should return TokenUsage, if LocalAI returns it (need to check it).

[ohadpinch]

Please raise priority

[langchain4j]

@ohadpinch it should have been fixed by https://github.com/langchain4j/langchain4j/pull/939/files, could you please check the latest version (or snapshot)?

[ohadpinch]

@ohadpinch it should have been fixed by https://github.com/langchain4j/langchain4j/pull/939/files, could you please check the latest version (or snapshot)?

I checked it, it's fixed with those changes.

[langchain4j]

@ohadpinch thank you!

[langchain4j]

Fixed by #939

[ohadpinch]

@langchain4j sorry for the mix up!
The issue still exist on #939, I ment I checked it with #942 and it seems solved.

[Kugaaa]

@Kugaaa I think AI Service should be changed to handle TokenUsage == null situation well, not all LLM providers return it.

Ideally, LocalAiChatModel should return TokenUsage, if LocalAI returns it (need to check it).

@langchain4j I think #939 not compatible with this situation

#942 If the solution is just return a null value, it appears that too little information is provided, so I want to use the null object mode return a NullTokenUsage, it will throw an exception and more infomation about used LLM not supported token usage.