Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ZEPPELIN-5973] Bump Livy 0.8.0-incubating #4678

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

pan3793
Copy link
Member

@pan3793 pan3793 commented Oct 16, 2023

What is this PR for?

Apache Livy 0.8.0-incubating was released at 2023-10-10, this PR aims to upgrade the built-in livy to the latest one.

What type of PR is it?

Improvement

Todos

  • - Task

What is the Jira issue?

ZEPPELIN-5973

How should this be tested?

Pass CI

Screenshots (if appropriate)

Questions:

  • Does the license files need to update? No.
  • Is there breaking changes for older versions? No.
  • Does this needs documentation? Yes, corresponding docs was updated

@pan3793 pan3793 changed the title [ZEPPELIN-5973] Bump Livy 0.8.0-incubating WIP [ZEPPELIN-5973] Bump Livy 0.8.0-incubating Oct 17, 2023
@jongyoul jongyoul marked this pull request as draft November 4, 2023 10:46
LIVY_VERSION="${1}"
SCALA_VERSION_SUFFIX=""
if [ $(version $LIVY_VERSION) -ge $(version "0.8.0") ]; then
SCALA_VERSION_SUFFIX="_${2}"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since 0.8.0, livy binary zip contains scala suffix

https://dist.apache.org/repos/dist/release/incubator/livy/0.7.1-incubating/apache-livy-0.7.1-incubating-bin.zip
https://dist.apache.org/repos/dist/release/incubator/livy/0.8.0-incubating/apache-livy-0.8.0-incubating_2.11-bin.zip

@@ -17,12 +17,20 @@
#

if [[ "$#" -ne 1 ]]; then
echo "usage) $0 [livy version]"
echo "usage) $0 <livy version> [scala version]"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

usually, <arg> represent the required arg, but [arg] represent the optional arg

@pan3793 pan3793 changed the title WIP [ZEPPELIN-5973] Bump Livy 0.8.0-incubating [ZEPPELIN-5973] Bump Livy 0.8.0-incubating Nov 7, 2023
@pan3793 pan3793 marked this pull request as ready for review November 7, 2023 18:34
@pan3793
Copy link
Member Author

pan3793 commented Nov 7, 2023

@jongyoul @Reamer this PR is ready for review, appreciate it if you have time to take a look.

@pan3793
Copy link
Member Author

pan3793 commented Nov 20, 2023

@jongyoul @zjffdu actually, we can refactor the test code of livy module by using a sub-process to call $LIVY_HOME/livy-server to manage the Livy server for testing, instead of pulling livy jars and invoking Java/Scala class directly.

I have a half-done PoC locally, if you don't mind, I will continue it when I have free time. Or if you think it's not a good idea, then let's keep it as-is.

@Reamer
Copy link
Contributor

Reamer commented Nov 20, 2023

I do not use livy. If there is a way to simplify the test execution we should take it. Currently the test execution is very specific.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants