Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a sleep during every scan round to avoid dead loop. #18557

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

xihuanbanku
Copy link

What changes are proposed in this pull request?

For big data tasks (eg, spark), might write a lot of files in the final stage. In that case, there will be a lot of write edge locks, which would trigger lockpool evict. All the locks have refcounts greater than 0, which means the locks cannot be evicted during that scan, and the lock pool size will keep what it is. There is no pause during each scan, and a dead loop accour. The server load grows up, and Alluxio master cannot offer service. So we add a wait during each scan to avoid this.

  • The following pic shows where dead loop occurs.
    image
  • The following pic shows server load grows very high.
    image
  • The jstack for Alluxio Master. (The nid matches java thread id.)
    image

For big data tasks (eg, spark), might write a lot of files in the final stage. In that case, there will be a lot of write edge locks, which would trigger lockpool evict. All the locks have refcounts greater than 0, which means the locks cannot be evicted during that scan. There is no pause during each scan, and a dead loop accour. The server load grows up. So we add a wait during each scan to avoid this.
@alluxio-bot
Copy link
Contributor

Thank you for your pull request.
In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement (CLA).
It's all electronic and will take just a few minutes. Please download CLA form here, sign, and e-mail back to cla@alluxio.org

@jasondrogba
Copy link
Contributor

Thanks for your contribution! could you sign a CLA please

@xihuanbanku
Copy link
Author

Thanks for your contribution! could you sign a CLA please

It's DONE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants