ETL

Architecture Diagram

Demo

Apache Superset dashboard credentials:

username: admin
password: admin

• Airbyte front-end: http://soufianeodf.tech:8000
• Apache Superset: http://soufianeodf.tech:8080

Requirements

• Docker

Getting Started

1- Clone repository

git clone https://github.com/soufianeodf/virtuocode-w1-etl.git etl

cd etl

2- Run docker compose in detached mode

docker-compose up -d

3- With your local superset container already running...

• 1- Setup your local admin account

  docker exec -it superset superset fab create-admin \
                 --username admin \
                 --firstname Superset \
                 --lastname Admin \
                 --email admin@superset.com \
                 --password admin
  

• 2- Migrate local DB to latest

  docker exec -it superset superset db upgrade
  

• 3- Load Examples (Optional)

  docker exec -it superset superset load_examples
  

• 4- Setup roles

  docker exec -it superset superset init
  

• 5- Install BigQuery Driver and restart superset container

  docker-compose exec superset pip install pybigquery
  docker-compose restart superset
  

Known issues

Github actions

Issue: Permission denied when "Deleting the contents of" when using actions/checkout

Solution:

sudo chown -R $USER:$USER /home/github/actions-runner/_work/{REPOSITORY_NAME_HERE}