Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix alluxio DoraMetaManger bug: update the listCache when load from ufs #18535

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix alluxio DoraMetaManger bug: update the listCache when load from ufs #18535

wants to merge 1 commit into from

Conversation

liiuzq-xiaobai
Copy link

What changes are proposed in this pull request?

When load fileStatus from UFS，not only updates the metaStore, but also updates the listCache.

Why are the changes needed?

There is an inconsistency problem with file metadata when operating "ls"

Does this PR introduce any user facing changes?

no

@alluxio-bot
Copy link
Contributor

Thank you for your pull request.
In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement (CLA).
It's all electronic and will take just a few minutes. Please download CLA form here, sign, and e-mail back to cla@alluxio.org

@liiuzq-xiaobai
Copy link
Author

alluxio-bot, check this please

@alluxio-bot
Copy link
Contributor

Automated checks report:

  • PR title follows the conventions: FAIL
    • The title of the PR does not pass all the checks. Please fix the following issues:
      • Title is too long (95 characters). Must be at most 72 characters.
      • First word of title ("Alluxio") is not an imperative verb. Please use one of the valid words
  • Commits associated with Github account: PASS

Some checks failed. Please fix the reported issues and reply
alluxio-bot, check this please
to re-run checks.

@liiuzq-xiaobai liiuzq-xiaobai changed the title Alluxio DoraMetaManger BugFix: invalidate or update the listCache when load fileStatus from UFS alluxio DoraMetaManger BugFix: invalidate or update the listCache Feb 29, 2024
@liiuzq-xiaobai
Copy link
Author

alluxio-bot, check this please

@alluxio-bot
Copy link
Contributor

Automated checks report:

  • PR title follows the conventions: FAIL
    • The title of the PR does not pass all the checks. Please fix the following issues:
      • Title must not end with whitespace
      • First word must be capitalized
      • First word of title ("alluxio") is not an imperative verb. Please use one of the valid words
  • Commits associated with Github account: PASS

Some checks failed. Please fix the reported issues and reply
alluxio-bot, check this please
to re-run checks.

@liiuzq-xiaobai liiuzq-xiaobai changed the title alluxio DoraMetaManger BugFix: invalidate or update the listCache fix alluxio DoraMetaManger bug: update the listCache when load from ufs Feb 29, 2024
@liiuzq-xiaobai
Copy link
Author

alluxio-bot, check this please

@alluxio-bot
Copy link
Contributor

Automated checks report:

  • PR title follows the conventions: FAIL
    • The title of the PR does not pass all the checks. Please fix the following issues:
      • First word must be capitalized
  • Commits associated with Github account: PASS

Some checks failed. Please fix the reported issues and reply
alluxio-bot, check this please
to re-run checks.

@liiuzq-xiaobai
Copy link
Author

alluxio-bot, check this please

@alluxio-bot
Copy link
Contributor

Automated checks report:

  • PR title follows the conventions: FAIL
    • The title of the PR does not pass all the checks. Please fix the following issues:
      • First word must be capitalized
  • Commits associated with Github account: PASS

Some checks failed. Please fix the reported issues and reply
alluxio-bot, check this please
to re-run checks.

@liiuzq-xiaobai liiuzq-xiaobai changed the title fix alluxio DoraMetaManger bug: update the listCache when load from ufs Fix alluxio DoraMetaManger bug: update the listCache when load from ufs Feb 29, 2024
@alluxio-bot
Copy link
Contributor

Automated checks report:

  • PR title follows the conventions: PASS
  • Commits associated with Github account: PASS

All checks passed!

YichuanSun
YichuanSun requested changes Mar 10, 2024
View reviewed changes
Copy link
Contributor

@YichuanSun YichuanSun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your work! I have an additional question: how do you find the inconsistent issue? By using alluxio In a real production environment or just by reading the code?

dora/core/server/worker/src/main/java/alluxio/worker/dora/DoraMetaManager.java
*/
public Optional<FileStatus> getFromUfs(String path) throws IOException {
public Optional<UfsStatus> getFromUfs(String path, UnderFileSystem ufs) throws IOException {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually alluxio is able to get ufs from an instance. I don't really understand why add ufs into paramter list.

dora/core/server/worker/src/main/java/alluxio/worker/dora/DoraMetaManager.java
Comment on lines +137 to +142
if (ufsStatus.isPresent() && ufsStatus.get().isFile()) {
if (mXAttrWriteToUFSEnabled) {
xattrMap = ufs.getAttributes(path);
}
fileStatus = PagedDoraWorker.buildFileStatusFromUfsStatus(
mCacheManager.getUsage(), ufs.getUnderFSType(), ufsStatus.get(), path, xattrMap);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why move the logic from getFromUfs to loadFromUfs?

@YichuanSun
Copy link
Contributor

btw, please first sign the CLA first #18535 (comment) before we approve your PR, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@YichuanSun YichuanSun YichuanSun requested changes

Assignees
No one assigned
Labels
cla-missing
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@liiuzq-xiaobai @alluxio-bot @YichuanSun