Steps to use

Prerequisites

• Hadoop 2.2 or later cluster or Sandbox.
• Apache Hive.
• Between 15 minutes and 2 days to generate data (depending on the Scale Factor you choose and available hardware).
• Have gcc in installed your system path. If your system does not have it, install it using yum or apt-get.

Clone

git clone https://github.com/kcheeeung/hive-testbench.git && cd hive-testbench/

Individual Steps

1. Build the benchmark

Build the benchmark you want to use (do all the prerequisites)

TPC-DS

./tpcds-build.sh

TPC-H

./tpch-build.sh

2. Generate the tables

Decide how much data you want. SCALE approximately is about # ~GB. Supported FORMAT includes: orc and parquet.

Generic Usage

nohup sh script.sh SCALE FORMAT

TPC-DS

nohup sh util_tablegentpcds.sh 10 orc

TPC-H

nohup sh util_tablegentpch.sh 10 orc

3. Run all the queries

• SCALE must be the SAME size from an existing database!
• Modify your settings in settings.sql.
• By default each query has a timeout it set to 2 hours! Change in util_internalRunQuery.sh where TIME_TO_TIMEOUT=120m.
• Run the queries!

TPC-DS Benchmark

nohup sh util_runtpcds.sh 10 orc

TPC-H Benchmark

nohup sh util_runtpch.sh 10 orc

Optional: Enable Performance Analysis Tool (PAT)

1. Connect Head to Worker Node

sh util_connect.sh YOURPASSWORD

2. Enable PAT

Go into util_runtpcds.sh or util_runtpch.sh. Switch the command by un/commenting. Example below.

# ./util_internalRunQuery.sh "$DATABASE" "$CURR_DIR$SETTINGS_PATH" "$CURR_DIR$query_path" "$CURR_DIR$LOG_PATH" "$i" "$CURR_DIR$REPORT_NAME.csv"

./util_internalGetPAT.sh /$CURR_DIR/util_internalRunQuery.sh "$DATABASE" "$CURR_DIR$SETTINGS_PATH" "$CURR_DIR$query_path" "$CURR_DIR$LOG_PATH" "$i" "$CURR_DIR$REPORT_NAME.csv" tpchPAT"$ID"/query"$i"/

Optional: Run Queries using Different Connection

Go into util_internalRunQuery.sh. Switch the command by uncommenting. Example below. Add the appropriate information (CLUSTERNAME and PASSWORD).

# timeout $TIME_TO_TIMEOUT beeline -u "jdbc:hive2://`hostname -f`:10001/$INTERNAL_DATABASE;transportMode=http" -i $INTERNAL_SETTINGSPATH -f $INTERNAL_QUERYPATH &>> $INTERNAL_LOG_PATH

timeout $TIME_TO_TIMEOUT beeline -u "jdbc:hive2://CLUSTERNAME.azurehdinsight.net:443/$INTERNAL_DATABASE;ssl=true;transportMode=http;httpPath=/hive2" -n admin -p PASSWORD -i $INTERNAL_SETTINGSPATH -f $INTERNAL_QUERYPATH &>> $INTERNAL_LOG_PATH

Troubleshooting

Did my X step finish?

Check the aaa_clock.txt or aab_clock.txt file.

ps -ef | grep .sh
ps -ef | grep beeline

Could not find database?

In the settings.sql file, add.

use DATABASENAME;

How to debug

Uncomment the following line.

# DEBUG_SCRIPT=X