Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The job stops restarting workers and exits if the traceback is a code bug. #1068

Open
workingloong opened this issue Apr 8, 2024 · 2 comments
Open

The job stops restarting workers and exits if the traceback is a code bug. #1068

workingloong opened this issue Apr 8, 2024 · 2 comments
Assignees
@BalaBalaYi
Labels
enhancement New feature or request question Further information is requested
Milestone
Backlog

Comments

@workingloong
Copy link
Collaborator

The restarted worker will fail again if the training fails due to a code bug. The job should exit as soon as possible to release resources on a cluster.
@BalaBalaYi BalaBalaYi self-assigned this Apr 22, 2024
@BalaBalaYi BalaBalaYi added bug Something isn't working question Further information is requested labels Apr 22, 2024
@BalaBalaYi
Copy link
Collaborator

A blacklist mechanism can be introduced to this case: throw an explicit error for user code errors with no more retry and free up resources.

@BalaBalaYi
Copy link
Collaborator

BalaBalaYi commented Apr 24, 2024 •
edited

user_code_bug_demo_log.txt

@BalaBalaYi BalaBalaYi added enhancement New feature or request and removed bug Something isn't working labels Apr 24, 2024
@BalaBalaYi BalaBalaYi added this to the Backlog milestone Jun 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@BalaBalaYi BalaBalaYi

Labels
enhancement New feature or request question Further information is requested
Projects
None yet
Milestone
Backlog
Development

When branches are created from issues, their pull requests are automatically linked.

1068-the-job-stops-restarting-workers-and-exits-if-the-traceback-is-a-code-bug intelligent-machine-learning/dlrover
2 participants
@workingloong @BalaBalaYi