Skip to content

Latest commit

 

History

History
143 lines (102 loc) · 3.43 KB

GETTING_STARTED.MD

File metadata and controls

143 lines (102 loc) · 3.43 KB
Raw

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

dbt init dbt_data_practitioner 

cd dbt_data_practitioner
touch profiles.yml

provide dbt with the connection details of ClickHouse instance.

dbt_data_practitioner:
  target: dev
  outputs:
    dev:
      type: clickhouse
      schema: sakila_db
      host: localhost
      port: 8123
      user: default
      password: root
      secure: False

Confirm whether dbt is able to connect to ClickHouse.

dbt debug

dbt docs generate

dbt docs serve

To access from your browser, navigate to: http://localhost:8080

Press Ctrl+C to exit.

cd models  
mkdir sakila_db  
cd sakila_db

actor_film_actor_join.sql defines our actual model. point_of_interest_1.sql is the model to be materialized as a view in ClickHouse.

touch actor_film_actor_join.sql
touch point_of_interest_1.sql
dbt build

Confirm the creation of the table actor_film_actor_join

SHOW CREATE TABLE sakila_db.actor_film_actor_join

You will see the table with the appropriate data types:

┌─statement────────────────────────────────────────────────────────────────────────────────┐
│ CREATE TABLE sakila_db.actor_film_actor_join
(
    `actor_actor_id` Int64,
    `first_name` Nullable(String),
    `last_name` Nullable(String),
    `actor_last_update` Nullable(DateTime64(3)),
    `film_actor_actor_id` Int64,
    `film_id` Int64,
    `film_actor_last_update` Nullable(DateTime64(3))
)
ENGINE = MergeTree
ORDER BY actor_actor_id
SETTINGS replicated_deduplication_window = 0, index_granularity = 8192 │
└──────────────────────────────────────────────────────────────────────────────────────────┘

We can now query the above table and view directly.

Querying this table, we can replicate the results of our earlier query with a simpler syntax:

SELECT * FROM sakila_db.actor_film_actor_join afaj  ORDER BY actor_actor_id

Snapshot - dbt snapshots allow a record to be made of changes to a mutable model over time. This in turn allows point-in-time queries on models, where analysts can “look back in time” at the previous state of a model.

  • We require a timestamp column to indicate record changes.
dbt snapshot.

Lineage Graph

cd dbt_data_practitioner
dagster-dbt project scaffold --project-name dagster_data_practitioner

cd dagster_data_practitioner
DAGSTER_DBT_PARSE_PROJECT_ON_LOAD=1 dagster dev

To access from your browser, navigate to: http://127.0.0.1:3000