Flaky test_memory_format with nn.BatchNorm2d when running with inductor #125967
Assignees
Labels
high priority
module: ci
Related to continuous integration
module: inductor
oncall: pt2
triaged
This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Comments
pytorchmergebot
pushed a commit
that referenced
this issue
May 11, 2024
Skipping the test in the context of #125967 until the issue is root caused and fixed properly. Pull Request resolved: #125970 Approved by: https://github.com/clee2000
|
Marking high priority as it is easily reproducible |
tinglvv
pushed a commit
to tinglvv/pytorch
that referenced
this issue
May 14, 2024
Skipping the test in the context of pytorch#125967 until the issue is root caused and fixed properly. Pull Request resolved: pytorch#125970 Approved by: https://github.com/clee2000
bdhirsh
added
triaged
|
I'll blindly suspect it's due to RNG state. I tried to repro, but this command: just shows: Add What's the correct way to run a disabled test? |
|
Oh, I skip the test in #125970 to keep trunk sane. Please revert my change before trying to reproduce it. Otherwise, it shows up as a skipped test. |
|
I at least figured out why test order matters here. It's due to dynamo compilation cache. If we run test for BatchNorm1D first, we already have a few compiled functions in dynamo cache. Later on when we test BatchNorm2D, the cache limit is reached, we fall back to eager and bypass the issue. To very the above point, I added torch._dynamo.reset() in the beginning of the test, now I can repro the issue even if BatchNorm2D is tested after BatchNorm1D. |
This was referenced May 17, 2024
shunting314
added a commit
that referenced
this issue
May 20, 2024
In #125967, we found test results depend on test order. The root cause is due to earlier tests populate dynamo cache and affect the later tests. This PR clear dynamo cache before each unit test so we get more deterministic result for unit test [ghstack-poisoned]
shunting314
added a commit
that referenced
this issue
May 20, 2024
In #125967, we found test results depend on test order. The root cause is due to earlier tests populate dynamo cache and affect the later tests. This PR clear dynamo cache before each unit test so we get more deterministic result for unit test [ghstack-poisoned]
shunting314
added a commit
that referenced
this issue
May 20, 2024
Fix #125967 . The test actually fail for empty 4D or 5D tensors when checking for memory format. I'm not exactly sure what recent inductor change cause the failure, but it may be not that important to maintain strides for an empty tensor. (?) I just skip the check for empty tensor. [ghstack-poisoned]
shunting314
added a commit
that referenced
this issue
May 20, 2024
Fix #125967 . The test actually fail for empty 4D or 5D tensors when checking for memory format. I'm not exactly sure what recent inductor change cause the failure, but it may be not that important to maintain strides for an empty tensor. (?) I just skip the check for empty tensor. [ghstack-poisoned]
shunting314
added a commit
that referenced
this issue
May 21, 2024
Fix #125967 . The test actually fail for empty 4D or 5D tensors when checking for memory format. I'm not exactly sure what recent inductor change cause the failure, but it may be not that important to maintain strides for an empty tensor. (?) I just skip the check for empty tensor. [ghstack-poisoned]
shunting314
added a commit
that referenced
this issue
May 21, 2024
In #125967, we found test results depend on test order. The root cause is due to earlier tests populate dynamo cache and affect the later tests. This PR clear dynamo cache before each unit test so we get more deterministic result for unit test [ghstack-poisoned]
shunting314
added a commit
that referenced
this issue
May 21, 2024
Fix #125967 . The test actually fail for empty 4D or 5D tensors when checking for memory format. I'm not exactly sure what recent inductor change cause the failure, but it may be not that important to maintain strides for an empty tensor. (?) I just skip the check for empty tensor. [ghstack-poisoned]
shunting314
added a commit
that referenced
this issue
May 21, 2024
In #125967, we found test results depend on test order. The root cause is due to earlier tests populate dynamo cache and affect the later tests. This PR clear dynamo cache before each unit test so we get more deterministic result for unit test [ghstack-poisoned]
pytorchmergebot
pushed a commit
that referenced
this issue
May 22, 2024
In #125967, we found test results depend on test order. The root cause is due to earlier tests populate dynamo cache and affect the later tests. This PR clear dynamo cache before each unit test so we get more deterministic result for unit test Pull Request resolved: #126586 Approved by: https://github.com/jansel
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This test starts to fail in trunk for
nn.BatchNorm2drecently. I think it's another example of #125239 where the test order of the tests matter.On devgpu, running the test alone fails like how it fails on CI:
But if I run it after
nn.BatchNorm1, it passes:cc @ezyang @gchanan @zou3519 @kadeng @msaroufim @seemethere @malfet @pytorch/pytorch-dev-infra @bdhirsh @anijain2305 @chauhang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @muchulee8 @ColinPeppler @amjames @desertfire
The text was updated successfully, but these errors were encountered: