Add getApplications API call and use it for Yarn app killing #3150
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tsangz2013
force-pushed
the
zhzeng/use-get-yarn-library-w-fallback
branch
from
9aaef0b
to
ecee390
Compare
tsangz2013
commented
Sep 9, 2022
| * @return the set of all to-be-killed (alive) yarn applications' IDs | ||
| * @throws IOException for RPC issue | ||
| * @throws YarnException for YARN server issue | ||
| */ |
There was a problem hiding this comment.
1: call yarn cluster to get app-report and transform to app-ids
tsangz2013
commented
Sep 9, 2022
| * @param jobProps should contain flow execution id, and the job log file's path | ||
| * @param log logger | ||
| * @return the set of application ids | ||
| */ |
There was a problem hiding this comment.
2: helper method first call yarnClient, with fallback to scan log
| * @throws IOException for RPC issue | ||
| * @throws YarnException for YARN server issue | ||
| */ | ||
| public static Set<String> getAllAliveAppIDsByExecID(final YarnClient yarnClient, |
There was a problem hiding this comment.
You would like to return the ApplicationReport from this method and create another method for application id. Since we need to get the user from ApplicationReport as well, if we write it in the current way, we need to have another call to get the user.
… tag and un-finished application states
…-scanning as fallback
tsangz2013
force-pushed
the
zhzeng/use-get-yarn-library-w-fallback
branch
from
ecee390
to
23ca091
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Follow up of #3145
Issue to solve: Upgrade of the yarn application killing logic in normal hadoop job cancel procedure
Changes made: Use yarnClient to get alive YARN application ids & use this method as first choice prior to log-scanning approach
Testing done: