Apache Spark

Spark Streaming, Machine Learning, Spark DataFrames and more.

Used platforms:

• Spark with pySpark (IDE: PyCharm | OS: Ubuntu)
• Databricks
• Jupyter Notebook
• AWS EC2 PySpark
• AWS EMR Cluster

---

Apache Spark & PySpark

• Spark and Python for Big Data with PySpark

Course topics and projects

Spark DataFrames Section Introduction

• Spark DataFrame Basics
• Spark DataFrame Operations
• GroupBy and Aggregate Functions
• Missing Data
• Dates and Timestamps
• Spark DataFrame Exercises.

Spark for Machine Learning.

• Linear Regression.
  • Projects:
    • Cruise ship.
    • E-commerce costumer expenditure - Description included.
• Logistic Regression.
  • Projects:
    • Churn predictor - Description included.
    • Titanic survival - Description included.
• Tree Methods.
  • Project:
    • Dog food spoil predictor - Description included.
• Clustering.
  • Projects:
    • Seeds clustering.
    • Start-up hack problem - Description included.
• Recommender System.
  • Project:
    • Movie rank predictor - Description included.
• Natural Language Processing.
  • Projects:
    • Spam_Detection_Filter - Description included.

Spark Streaming:

• Projects:
  • Spark_Streaming_Twitter_Project - Description included.

Acquired skills:

• Utilizing Spark 2.0 DataFrame Syntax to analyze big data.
• Engaging in consulting projects that simulate real-world scenarios.
• Applying logistic regression to classify customer churn.
• Utilizing Random Forests and Gradient Boosted Trees in Spark for classification.
• Creating powerful machine learning models with Spark's MLlib.
• Gaining familiarity with the DataBricks platform.
• Setting up Amazon Web Services EC2 for big data analysis.
• Utilizing AWS Elastic MapReduce Service.
• Leveraging the power of Linux in a Spark environment.
• Developing a spam filter using Spark and natural language processing.
• Analyzing tweets in real-time with Spark Streaming.