Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] compare.py benchmark fails #770

Open
jrp2014 opened this issue Mar 2, 2024 · 0 comments
Open

[BUG] compare.py benchmark fails #770

jrp2014 opened this issue Mar 2, 2024 · 0 comments

Comments

@jrp2014
Copy link

jrp2014 commented Mar 2, 2024

Running the compare benchmark from benchmarks/python/comparative leads to failure.

> python compare
:
0.5914489247792871	celu --size 32x16x1024 --cpu
0.6487723472573234	log_sigmoid --size 32x16x1024
-0.2201783376280044	log_sigmoid --size 32x16x1024 --cpu
Traceback (most recent call last):
  File "/Users/jrp/Documents/AI/mlx/mlx/benchmarks/python/comparative/compare.py", line 17, in run_or_raise
    return float(result.stdout)
           ^^^^^^^^^^^^^^^^^^^^
ValueError: could not convert string to float: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jrp/Documents/AI/mlx/mlx/benchmarks/python/comparative/compare.py", line 214, in <module>
    compare_filtered("step --size 32x16x1024")
  File "/Users/jrp/Documents/AI/mlx/mlx/benchmarks/python/comparative/compare.py", line 89, in <lambda>
    compare_filtered = lambda x: compare(x.split() + rest) if _filter(x) else None
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jrp/Documents/AI/mlx/mlx/benchmarks/python/comparative/compare.py", line 24, in compare
    t_torch = run_or_raise(["python", BENCH_TORCH] + args)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jrp/Documents/AI/mlx/mlx/benchmarks/python/comparative/compare.py", line 19, in run_or_raise
    raise ValueError(f"stdout: {result.stdout}\nstderr: {result.stderr}")
ValueError: stdout: b''
stderr: b'Traceback (most recent call last):\n  File "/Users/jrp/Documents/AI/mlx/mlx/benchmarks/python/comparative/bench_torch.py", line 450, in <module>\n    raise ValueError("Unknown benchmark")\nValueError: Unknown benchmark\n'

Expected behavior
The benchmark should run to conclusion.

Desktop (please complete the following information):

  • MacOS 14.3.1
  • Head of mlx repo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant