Skip to content

Java client for submitting a remote job to HDInsight Spark cluster via Livy.

License

Notifications You must be signed in to change notification settings

kojish/hdinsight-spark-livy-client

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

36 Commits
 
 
 
 
 
 
 
 

Repository files navigation

hdinsight-spark-livy-client

Java client for submitting a remote job to HDInsight Spark cluster.

Introduction

The client supports to send a batch job and create an interactive session with Spark. The implmentation is based on the specification described in the following documents:

Build Prerequisites

You will need to download and install the following to build the code:

  • JDK 1.8 or later
  • Maven 3.x

Dependency

TBD.

Sample

The sample codes are included in the src/main/java/sample directory. The sample shows you how to use the client for both batch and interactive.

Quick Start

Here is how to run the sample application, assuming you already have installed Java 8+ and Maven 3.x in your environment.

1. First of all, you need to set a username, password, and endpoint to get an access to your 
   hdinsight cluster. Open InteractiveSample2.java that is in src/main/java/sample directory, 
   and set username, password, and endpoint name to LivyInteractiveClient() class.
2. Change the directory to hdinsight-spark-livy-client/livy-client. You can find pom.xml, and 
   run the follwoiong command for compiling and building jar file.
   >mvn clean package -DskipTests=true
3. Run the application with the following command.
   >java -cp ./target/livy-client-0.0.1-SNAPSHOT.jar sample/Main

About

Java client for submitting a remote job to HDInsight Spark cluster via Livy.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages