Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix fuse sync issue between readdir and release #18511

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

secfree
Copy link
Contributor

@secfree secfree commented Feb 6, 2024

What changes are proposed in this pull request?

Fix fuse sync issue between readdir and release.

Why are the changes needed?

The following test failed under an alluxio-fuse mount point

$ cat test.sh
mkdir dir.01
echo hello > dir.01/file.01
rm -rf dir.01

$ bash test.sh
rm: cannot remove 'dir.01': Input/output error

However, executing these commands manually in sequence is ok.

Checked the log and found

2024-02-06 09:24:00,555 DEBUG [Thread-7](AlluxioFuseUtils.java:651) - Enter: Fuse.Release(path=/dir.01/file.01,fd=7)
2024-02-06 09:24:00,556 DEBUG [Thread-17](AlluxioFuseUtils.java:655) - Exit (0): Fuse.Getattr(path=/) in 1 ms
2024-02-06 09:24:00,556 DEBUG [Thread-2](AlluxioFuseUtils.java:651) - Enter: Fuse.Flush(path=/test.sh,fd=6)
2024-02-06 09:24:00,556 DEBUG [Thread-2](AlluxioFuseUtils.java:655) - Exit (0): Fuse.Flush(path=/test.sh,fd=6) in 0 ms
2024-02-06 09:24:00,557 DEBUG [Thread-11](AlluxioFuseUtils.java:651) - Enter: Fuse.Getattr(path=/dir.01)
2024-02-06 09:24:00,558 DEBUG [Thread-11](AlluxioFuseUtils.java:655) - Exit (0): Fuse.Getattr(path=/dir.01) in 1 ms
2024-02-06 09:24:00,558 DEBUG [Thread-2](AlluxioFuseUtils.java:651) - Enter: Fuse.Readdir(path=/dir.01)
2024-02-06 09:24:00,559 DEBUG [Thread-2](AlluxioFuseUtils.java:655) - Exit (0): Fuse.Readdir(path=/dir.01) in 1 ms
2024-02-06 09:24:00,567 DEBUG [Thread-7](AlluxioFuseUtils.java:655) - Exit (0): Fuse.Release(path=/dir.01/file.01,fd=7) in 12 ms
2024-02-06 09:24:00,567 DEBUG [Thread-2](AlluxioFuseUtils.java:651) - Enter: Fuse.Rmdir(path=/dir.01)
2024-02-06 09:24:00,572 WARN  [Thread-2](HdfsUnderFileSystem.java:842) - Attempt count 1 : org.apache.hadoop.fs.PathIsNotEmptyDirectoryException: `o3fs://alluxio.test.dev/dir.01': Directory is not empty
...
2024-02-06 09:24:00,587 ERROR [Thread-2](AlluxioFuseUtils.java:675) - Failed to Fuse.Rmdir(path=/dir.01) with unexpected throwable:
alluxio.exception.runtime.UnknownRuntimeException: `o3fs://alluxio.test.dev/dir.01': Directory is not empty
        at alluxio.exception.runtime.AlluxioRuntimeException.from(AlluxioRuntimeException.java:207)
        at alluxio.client.file.ufs.UfsBaseFileSystem.call(UfsBaseFileSystem.java:493)
        at alluxio.client.file.ufs.UfsBaseFileSystem.delete(UfsBaseFileSystem.java:196)
        at alluxio.client.file.MetadataCachingFileSystem.delete(MetadataCachingFileSystem.java:122)
        at alluxio.client.file.FileSystem.delete(FileSystem.java:298)
        at alluxio.fuse.AlluxioJniFuseFileSystem.rmInternal(AlluxioJniFuseFileSystem.java:480)
        at alluxio.fuse.AlluxioJniFuseFileSystem.lambda$rmdir$14(AlluxioJniFuseFileSystem.java:463)
        at alluxio.fuse.AlluxioFuseUtils.call(AlluxioFuseUtils.java:653)
        at alluxio.fuse.AlluxioJniFuseFileSystem.rmdir(AlluxioJniFuseFileSystem.java:463)
        at alluxio.jnifuse.AbstractFuseFileSystem.rmdirCallback(AbstractFuseFileSystem.java:314)
Caused by: org.apache.hadoop.fs.PathIsNotEmptyDirectoryException: `o3fs://alluxio.test.dev/dir.01': Directory is not empty
        at org.apache.hadoop.fs.ozone.BasicOzoneFileSystem$DeleteIterator.<init>(BasicOzoneFileSystem.java:466)
        at org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.innerDelete(BasicOzoneFileSystem.java:491)
        at org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.delete(BasicOzoneFileSystem.java:540)
        at alluxio.underfs.hdfs.HdfsUnderFileSystem.delete(HdfsUnderFileSystem.java:826)
        at alluxio.underfs.hdfs.HdfsUnderFileSystem.deleteDirectory(HdfsUnderFileSystem.java:355)
        at alluxio.underfs.UnderFileSystemWithLogging$10.call(UnderFileSystemWithLogging.java:272)
        at alluxio.underfs.UnderFileSystemWithLogging$10.call(UnderFileSystemWithLogging.java:269)
        at alluxio.underfs.UnderFileSystemWithLogging.call(UnderFileSystemWithLogging.java:1302)
        at alluxio.underfs.UnderFileSystemWithLogging.deleteDirectory(UnderFileSystemWithLogging.java:269)
        at alluxio.client.file.ufs.UfsBaseFileSystem.lambda$delete$3(UfsBaseFileSystem.java:206)
        at alluxio.client.file.ufs.UfsBaseFileSystem.call(UfsBaseFileSystem.java:491)
        ... 8 more

The log shows that the cause is "Enter: Fuse.Readdir") happened between "Enter: Fuse.Release" and "Exit: Fuse.Release"

2024-02-06 09:24:00,555 DEBUG [Thread-7](AlluxioFuseUtils.java:651) - Enter: Fuse.Release(path=/dir.01/file.01,fd=7)
...
2024-02-06 09:24:00,558 DEBUG [Thread-2](AlluxioFuseUtils.java:651) - Enter: Fuse.Readdir(path=/dir.01)
2024-02-06 09:24:00,559 DEBUG [Thread-2](AlluxioFuseUtils.java:655) - Exit (0): Fuse.Readdir(path=/dir.01) in 1 ms
2024-02-06 09:24:00,567 DEBUG [Thread-7](AlluxioFuseUtils.java:655) - Exit (0): Fuse.Release(path=/dir.01/file.01,fd=7) in 12 ms

As I was using Ozone as UFS and connected to it directly, and an Ozone key is not visible before the commit action (Release in Alluxio), these called the issue.

After applying this PR, the test works well.

Does this PR introduce any user facing changes?

NO

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant