Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problems with non-Open AI models #462

Open
rbyh opened this issue May 9, 2024 · 3 comments
Open

Problems with non-Open AI models #462

rbyh opened this issue May 9, 2024 · 3 comments
Assignees
Labels
bug Something isn't working

Comments

@rbyh
Copy link
Contributor

rbyh commented May 9, 2024

I originally thought this was a problem with scenarios, but non-scenario questions are not running either:

from edsl.questions import QuestionMultipleChoice
from edsl import Model

model = Model('claude-3-opus-20240229')

q = QuestionMultipleChoice.example()

results = q.by(model).run()
image
@rbyh rbyh added the bug Something isn't working label May 9, 2024
@rbyh
Copy link
Contributor Author

rbyh commented May 9, 2024

image

@rbyh
Copy link
Contributor Author

rbyh commented May 9, 2024

image

@rbyh
Copy link
Contributor Author

rbyh commented May 9, 2024

┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
┃ Question     ┃                                  ┃                   ┃                      ┃
┃ name         ┃ Exception                        ┃              Time ┃ Traceback            ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩
│ how_feeling  │ Exception("Deep Infra response   │ 1715214725.213443 │                      │
│              │ does not contain 'results' key:  │                   │                      │
│              │ {'detail': 'User is not          │                   │                      │
│              │ authorized to access this        │                   │                      │
│              │ resource', 'elapsed_time':       │                   │                      │
│              │ 0.25301194190979004,             │                   │                      │
│              │ 'timestamp': 1715214725.213269,  │                   │                      │
│              │ 'cached_response': False,        │                   │                      │
│              │ 'cache_key':                     │                   │                      │
│              │ '2238a07cda21600f9dab7b19bab304… │                   │                      │
│ how_feeling  │ Exception("Deep Infra response   │  1715214726.22962 │                      │
│              │ does not contain 'results' key:  │                   │                      │
│              │ {'detail': 'User is not          │                   │                      │
│              │ authorized to access this        │                   │                      │
│              │ resource', 'elapsed_time':       │                   │                      │
│              │ 0.0021572113037109375,           │                   │                      │
│              │ 'timestamp': 1715214726.2294662, │                   │                      │
│              │ 'cached_response': True,         │                   │                      │
│              │ 'cache_key': None}")             │                   │                      │
│ how_feeling  │ Exception("Deep Infra response   │ 1715214728.247215 │                      │
│              │ does not contain 'results' key:  │                   │                      │
│              │ {'detail': 'User is not          │                   │                      │
│              │ authorized to access this        │                   │                      │
│              │ resource', 'elapsed_time':       │                   │                      │
│              │ 0.0029137134552001953,           │                   │                      │
│              │ 'timestamp': 1715214728.2469828, │                   │                      │
│              │ 'cached_response': True,         │                   │                      │
│              │ 'cache_key': None}")             │                   │                      │
│ how_feeling  │ Exception("Deep Infra response   │ 1715214732.265152 │                      │
│              │ does not contain 'results' key:  │                   │                      │
│              │ {'detail': 'User is not          │                   │                      │
│              │ authorized to access this        │                   │                      │
│              │ resource', 'elapsed_time':       │                   │                      │
│              │ 0.002839803695678711,            │                   │                      │
│              │ 'timestamp': 1715214732.264988,  │                   │                      │
│              │ 'cached_response': True,         │                   │                      │
│              │ 'cache_key': None}")             │                   │                      │
│ how_feeling  │ Exception("Deep Infra response   │ 1715214740.295725 │                      │
│              │ does not contain 'results' key:  │                   │                      │
│              │ {'detail': 'User is not          │                   │                      │
│              │ authorized to access this        │                   │                      │
│              │ resource', 'elapsed_time':       │                   │                      │
│              │ 0.009119033813476562,            │                   │                      │
│              │ 'timestamp': 1715214740.295447,  │                   │                      │
│              │ 'cached_response': True,         │                   │                      │
│              │ 'cache_key': None}")             │                   │                      │
│ how_feeling  │ InterviewErrorPriorTaskCanceled… │ 1715214740.297733 │                      │
│              │ tasks failed for how_feeling')   │                   │                      │
└──────────────┴──────────────────────────────────┴───────────────────┴──────────────────────┘

@rbyh rbyh changed the title Scenarios are only working with OpenAI models Problems with non-Open AI models May 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants