What is the best testing framework to use with instructor?

[daver987]

Hello everyone, I was wondering what other people use as a testing framework for prompts etc. when you use instructor?
I am finding that most are either way to complicated, don't support function/tool calls or if they do is a second class citizen. Example, promptfoo seemed promising but, I don't want to build my pydantic models, then create separate json schema that has to go inside of yaml, its messy and defeats the purpose of a test. It seems like everyone forgot about function calls, yet what you can do with them is amazing.

[jxnl]

pytest

[jxnl]

Makes sense.

I would challenge you to think harder about how to break down the code itself to be more testable.

[daver987]

I've broken it down some, but not enough. The example in the docs of the cache function was a bit of a lightbulb moment for code reuse. I'm normally working with JS/TS and I don't really ever use classes. I'm going to work on this and see if I can get it broken down better. Thank you for the input, I appreciate it greatly.

[jxnl]

There's a JavaScript version actually!

https://instructor-ai.github.io/instructor-js/

[daver987]

Nice! That's going to be a challenge and a half to replicate the python version. I have to say, I really like Pydantic a lot more than zod. I mean zod is pretty awesome, but I feel that Pydantic is more intuitive and more fine grained for sure, then add that to FastApi, I may never use server routes again. LOL.

Thank You again for you help, i was able to achieve everything I needed to with pytest, I got some good ideas from the examples and boom, no need for some extra dependency. I have a nice setup that allows me to test the open source along with the openai models if I want, which is nice. It works great. I'm so glad I saw your talk back in October, it really allowed me to think different about how to use AI, and because of that I was given a chance at my dream job, and now it will hopefully become a permanent position.