-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LangSmith "Criteria()" and "LabeledCriteria()" do not accept custom criteria in TypeScript, despite examples #5188
Comments
It looks like the issue you're encountering with For custom criteria in TypeScript, you should leverage the import { CriteriaEvalChain } from '@langchain/evaluation/criteria';
import { BaseLanguageModelInterface } from '@langchain/core/language_models/base';
const llm: BaseLanguageModelInterface = /* your language model instance */;
const customCriterion = {
creativity: "Does the submission demonstrate novelty or unique ideas?",
};
const criteriaEvalChain = await CriteriaEvalChain.fromLLM(llm, customCriterion);
const input = "Tell me a joke";
const prediction = "I ate some square pie but I don't know the square of pi.";
const res = await criteriaEvalChain.evaluateStrings({ input, prediction });
console.log(res); This example demonstrates defining a Remember, the If you're still facing issues or have more questions, feel free to ask!
|
@dosu This doesn't apply in this scenario -- I don't want to create an eval chain, I need to create an EvalConfig, since that is what |
To address the issue with using custom criteria in an
const customCriteriaConfig = Criteria("helpfulness", {
formatEvaluatorInputs: ({ rawInput, rawPrediction }) => ({
prediction: getSingleStringifiedValue(rawPrediction),
input: getSingleStringifiedValue(rawInput),
}),
llm: /* your language model instance */,
feedbackKey: "customFeedbackKey" // Optional: specify if needed
}); In this setup,
const evalConfig = {
evaluators: [customCriteriaConfig],
// Include other necessary configuration options here
}; This configuration can then be used directly with By following these steps, you should be able to use custom criteria within your
|
Sorry for the delay! @dqbd could you have a look at this one? |
Hello @w-biggs! The issue stems from the invalid type, which will be fixed in #5403. In the meantime, you can either:
{
evaluators: [
// @ts-expect-error Invalid type
Criteria({
isCompliant: "Does the submission comply with the requirements of XYZ",
}),
],
}
|
Checked other resources
Example Code
Neither of these work. The first is what is implied to work by the comments in
langchainjs/langchain/src/smith/config.ts
Line 219 in b9d86b1
Error Message and Stack Trace (if applicable)
No response
Description
There seems to be no way to use custom criteria in an criteria evaluator, which differs from the Python implementation.
The first method above does not seem to work -- the object is not compatible with what it wants to be in the evaluators array. The second method does not work either --
Criteria()
's argument is aCriteria
type instead of aCriteriaLike
, and if that line is@ts-ignore
'd, it seems to fail with a 422 "expected a string" error when sending "feedback" to the LangSmith server.Putting this issue in here because the code is in this repo and not in
LangSmith
proper (yet?)System Info
langchain@0.1.35
Node v21.5.0
The text was updated successfully, but these errors were encountered: