Data Science in Apache Py-Spark

Customer Churn Project

Ensemble Models

Level: Moderate

Language: Python

Requirements:

• HDP + CDSW
• Spark 2.3

Author: Ian Brooks

Follow: LinkedIn - Ian Brooks PhD

Orginal Fork From: CDSW Demo

Churn Baby Churn

This Github repo is designed to be optmized for Cloudera Data Science Workbench (CDSW), but it is not required. The PySpark code can be used with Apache Spark, and the code examples will run with the included dataset.

In this project, there are 5 different supervised classifer models designed for telco customer churn. The first four classsifer models user are: Random Forest, Gradient Boost Tree, Suport Vector Machines, and Multilayer Perception. The most sucessful model is a Stacked Ensemble Model.

CDSW Run Instructions

1. In CSDW, download the project using the git url for here
2. Open a new session, and execute the setup.sh file
3. In Experiments, run the following scripts
   • dsforteko_pyspark.py - vanilla random forest churn model
   • gbt_churn_pyspark.py - gradient boost tree churn model with normamlized variables, hyperturning, and crossvalidation
   • mlp_churn_pyspark.py - multilayer perceptron churn model with normamlized variables, hyperturning, and crossvalidatio
   • rf_churn_pyspark.py - random forest churn model with normamlized variables, hyperturning, and crossvalidation
   • svm_churn_pyspark.py - support vection machine churn model with normamlized variables, hyperturning, and crossvalidation
4. Once all experiments have completed, the stacked ensemble classifer model be built, run the following script to build the stacked model.
   • stacked_churn_pyspark.py - stacked ensemble model trained on the prediction of random forest, gradient boost tree, and support vector machine
5. Once the stacked model has been built, the stacked model can be deployed using the following script.
   • predict_churn_stackedMLP_pyspark.py