Skip to content

Issues: symflower/eval-dev-quality

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

24 Open 23 Closed
24 Open 23 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Follow-Up from using Git to reset the temporary directory refactor Make code cleaner/better
#141 opened May 22, 2024 by Munsio v0.5.0
@zimmski
Test for pulling Ollama model is flaky bug Something isn't working
#135 opened May 18, 2024 by zimmski v0.5.0
2
Track how many characters were present in code part / complete response enhancement New feature or request
#128 opened May 17, 2024 by bauersimon v0.5.0
@bauersimon
Exclude openrouter/auto since it is just a random model bug Something isn't working
#126 opened May 17, 2024 by bauersimon v0.5.0
@zimmski
Preload/Unload Ollama models before prompting enhancement New feature or request
#116 opened May 15, 2024 by Munsio
4 tasks done
v0.5.0
@Munsio @bauersimon
5
Generic OpenAI API provider enhancement New feature or request
#111 opened May 14, 2024 by bauersimon
5 of 7 tasks
v0.5.0
@bauersimon
4
Flaky CI because of corrupted Z3 installation bug Something isn't working OS:MacOS OS:Windows
#107 opened May 13, 2024 by bauersimon v0.5.0
@bauersimon
Follow up: Ollama Support enhancement New feature or request
#100 opened May 8, 2024 by bauersimon
5 of 12 tasks
v0.5.0
@bauersimon
InstallToolsPath is not used for test execution (make test) bug Something isn't working
#93 opened May 6, 2024 by Munsio v0.5.0
5
Include metrics about the models for comparing models enhancement New feature or request
#82 opened Apr 28, 2024 by zimmski
7 tasks
v0.5.0
Add linters where each error is a metric enhancement New feature or request
#81 opened Apr 28, 2024 by zimmski v0.5.0
1
Introduce an AST-differ that also gives metrics enhancement New feature or request
#80 opened Apr 28, 2024 by zimmski
5 tasks
v0.5.0
3
Roadmap for v0.5.0 enhancement New feature or request
#79 opened Apr 26, 2024 by zimmski 100+ v0.5.0
@zimmski
2
Fix svg Y axis ticks bug Something isn't working
#73 opened Apr 25, 2024 by bauersimon v0.5.0
@bauersimon
4
Infer if a model produced "too much" code enhancement New feature or request
#44 opened Apr 18, 2024 by bauersimon v0.5.0
Infer if a model actually returned source code enhancement New feature or request
#43 opened Apr 18, 2024 by bauersimon v0.5.0
Implement "chain of thought" tasks enhancement New feature or request help wanted Extra attention is needed
#31 opened Apr 15, 2024 by zimmski v0.5.0
Retry with feedback and retry without feedback. enhancement New feature or request
#30 opened Apr 11, 2024 by rnbwdsh v0.5.0
Better prompting with templates: Setting a mandatory document start enhancement New feature or request
#29 opened Apr 11, 2024 by rnbwdsh v0.5.0
Better error handling - panic instead of "failing up" bug Something isn't working
#28 opened Apr 11, 2024 by rnbwdsh v0.5.0
@zimmski
1
Do an up-to-date leaderboard/dashboard for current models current evaluation enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
#26 opened Apr 10, 2024 by zimmski v0.5.0
Upload binaries of the evaluation binary for all OSes and architectures for users that only want to benchmark enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
#20 opened Apr 4, 2024 by zimmski v0.5.0
Include linters in the development environment and CI enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
#18 opened Apr 4, 2024 by zimmski v0.5.0
Sandbox execution enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
#17 opened Apr 4, 2024 by zimmski v0.5.0
ProTip! Mix and match filters to narrow down what you’re looking for.