structured-generation-benchmark

To use Large Language Models (LLMs) effectively and reliably, it's essential to include structured generation techniques. Being able to get outputs like regular expressions, JSON, or a Pydantic data model is key for making useful software.

But what's the real effect of using libraries like Outlines or Instructor to achieve that goal?

This repository has put together evaluations to answer this question.

Function Calling

The ability of the LLM to call functions.

Datasets

• Berkeley Function Calling Leaderboard [April 16, 2024 update]

Evaluation

• We deployed a modal function to run open-source models using Transformers + Outlines.
• We created different model handlers to run the Gorilla BFCL scripts [April 6, 2024 version] for the AST simple evaluation category.
• We evaluated and reported the results comparing them with the Leaderboard Website [April 26, 2024 version].

Reports

• Outlines Function Calling Evaluation
• Instructor Function Calling Evaluation