Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[inductor, 3.12] test_torchinductor.py test_config_option_dont_assume_alignment_cuda failing in 3.12 #125979

Closed
williamwen42 opened this issue May 10, 2024 · 2 comments
Closed

[inductor, 3.12] test_torchinductor.py test_config_option_dont_assume_alignment_cuda failing in 3.12 #125979

williamwen42 opened this issue May 10, 2024 · 2 comments
Assignees
@williamwen42
Labels
module: inductor oncall: pt2 triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@williamwen42
Copy link
Member

williamwen42 commented May 10, 2024 •
edited by pytorch-bot bot

Repro: python -m pytest test/inductor/test_torchinductor.py -k test_config_option_dont_assume_alignment

Error:

Traceback (most recent call last):
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/unittest/case.py", line 634, in run
    self._callTestMethod(testMethod)
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/unittest/case.py", line 589, in _callTestMethod
    if method() is not None:
       ^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/testing/_internal/common_utils.py", line 2763, in wrapper
    method(*args, **kwargs)
  File "/data/users/williamwen/pytorch/test/inductor/test_torchinductor.py", line 9843, in new_test
    return value(self)
           ^^^^^^^^^^^
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/test/inductor/test_torchinductor.py", line 9151, in test_config_option_dont_assume_alignment
    res = fn_c(inp)
          ^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/eval_frame.py", line 420, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/convert_frame.py", line 981, in catch_errors
    return callback(frame, cache_entry, hooks, frame_state, skip=1)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/convert_frame.py", line 822, in _convert_frame
    result = inner_convert(
             ^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/convert_frame.py", line 410, in _convert_frame_assert
    return _compile(
           ^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_utils_internal.py", line 70, in wrapper_function
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/convert_frame.py", line 703, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/utils.py", line 273, in time_wrapper
    r = func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/convert_frame.py", line 570, in compile_inner
    out_code = transform_code_object(code, transform)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/bytecode_transformation.py", line 1167, in transform_code_object
    transformations(instructions, code_options)
  File "/data/users/williamwen/pytorch/torch/_dynamo/convert_frame.py", line 172, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/convert_frame.py", line 517, in transform
    tracer.run()
  File "/data/users/williamwen/pytorch/torch/_dynamo/symbolic_convert.py", line 2234, in run
    super().run()
  File "/data/users/williamwen/pytorch/torch/_dynamo/symbolic_convert.py", line 884, in run
    while self.step():
          ^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/symbolic_convert.py", line 799, in step
    self.dispatch_table[inst.opcode](self, inst)
  File "/data/users/williamwen/pytorch/torch/_dynamo/symbolic_convert.py", line 2424, in RETURN_VALUE
    self._return(inst)
  File "/data/users/williamwen/pytorch/torch/_dynamo/symbolic_convert.py", line 2409, in _return
    self.output.compile_subgraph(
  File "/data/users/williamwen/pytorch/torch/_dynamo/output_graph.py", line 1079, in compile_subgraph
    self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/output_graph.py", line 1296, in compile_and_call_fx_graph
    compiled_fn = self.call_user_compiler(gm)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/utils.py", line 273, in time_wrapper
    r = func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/output_graph.py", line 1387, in call_user_compiler
    raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
  File "/data/users/williamwen/pytorch/torch/_dynamo/output_graph.py", line 1368, in call_user_compiler
    compiled_fn = compiler_fn(gm, self.example_inputs())
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/repro/after_dynamo.py", line 127, in debug_wrapper
    compiled_gm = compiler_fn(gm, example_inputs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/__init__.py", line 1747, in __call__
    return compile_fx(model_, inputs_, config_patches=self.config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_inductor/compile_fx.py", line 1478, in compile_fx
    return aot_autograd(
           ^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/backends/common.py", line 65, in compiler_fn
    cg = aot_module_simplified(gm, example_inputs, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_functorch/aot_autograd.py", line 958, in aot_module_simplified
    compiled_fn = create_aot_dispatcher_function(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/utils.py", line 273, in time_wrapper
    r = func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_functorch/aot_autograd.py", line 685, in create_aot_dispatcher_function
    compiled_fn = compiler_fn(
                  ^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 470, in aot_wrapper_dedupe
    return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 672, in aot_wrapper_synthetic_base
    return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 169, in aot_dispatch_base
    compiled_fw = compiler(fw_module, updated_flat_args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/utils.py", line 273, in time_wrapper
    r = func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_inductor/compile_fx.py", line 1382, in fw_compiler_base
    return inner_compile(
           ^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/repro/after_aot.py", line 83, in debug_wrapper
    inner_compiled_fn = compiler_fn(gm, example_inputs)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_inductor/debug.py", line 304, in inner
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/utils.py", line 273, in time_wrapper
    r = func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_inductor/compile_fx.py", line 498, in compile_fx_inner
    compiled_graph = FxGraphCache.load(
                     ^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_inductor/codecache.py", line 987, in load
    compiled_graph = compile_fx_fn(gm, example_inputs, **fx_kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_inductor/compile_fx.py", line 803, in fx_codegen_and_compile
    compiled_fn = graph.compile_to_fn()
                  ^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_inductor/graph.py", line 1707, in compile_to_fn
    return self.compile_to_module().call
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_dynamo/utils.py", line 273, in time_wrapper
    r = func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_inductor/graph.py", line 1654, in compile_to_module
    mod = PyCodeCache.load_by_key_path(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_inductor/codecache.py", line 2570, in load_by_key_path
    mod = _reload_python_module(key, path)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_inductor/runtime/compile_tasks.py", line 44, in _reload_python_module
    exec(code, mod.__dict__, mod.__dict__)
  File "/tmp/tmpobj5fvaf/e5/ce5zelh5uynccszfoglj5purhekltth24mypaeyuv5fo3lik3rjx.py", line 71, in <module>
  File "/data/users/williamwen/pytorch/torch/_inductor/codecache.py", line 3126, in wait
    scope[key] = result.result()
                 ^^^^^^^^^^^^^^^
  File "/data/users/williamwen/pytorch/torch/_inductor/codecache.py", line 2928, in result
    self.future.result()
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
OSError: could not get source code

cc @ezyang @msaroufim @bdhirsh @anijain2305 @chauhang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire
@williamwen42
Copy link
Member Author

cc @masnesral - appears to be erroring in the inductor code cache.

@bdhirsh bdhirsh added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label May 13, 2024
@williamwen42
Copy link
Member Author

This will be fixed by re-enabling AsyncCompile.warmpool: #124192.

@williamwen42 williamwen42 mentioned this issue May 20, 2024
Closed
pytorchmergebot pushed a commit that referenced this issue May 21, 2024
@williamwen42 @pytorchmergebot
[3.12, inductor] re-enable AsyncCompile.warm_pool for 3.12 (#126724)
b36e018 
Somehow working now? Fixes #124192 and #125979.

Still getting the warning
```
/home/williamwen/local/installs/python3.12/debug/install/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=2360707) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
```
though

Pull Request resolved: #126724
Approved by: https://github.com/masnesral, https://github.com/jansel
@williamwen42 williamwen42 self-assigned this May 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@williamwen42 williamwen42

Labels
module: inductor oncall: pt2 triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bdhirsh @williamwen42