Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allows naming conventions to be changed #998

Draft
wants to merge 37 commits into
base: devel
Choose a base branch
from

Conversation

rudolfix
Copy link
Collaborator

@rudolfix rudolfix commented Feb 25, 2024

Description

This PR attempts to allow any naming convention to be used (ie UPPER CASE or CamelCase). Until now any naming convention that changed casing of columns was failing. Also destinations have problems when reflecting such columns from information schema.

todo:

  1. Make sure dlt engine is agnostic to the naming convention used on schema, table and column level (done, not well tested)
  2. Make sure that schema settings (preferred types, normalizer settings, default hints) are also updated in (1)
  3. Expose and document methods to change default hints, preferred settings and other compiled properties, update documentation
  4. Make sure that all destinations work with forced lower and upper case naming conventions (not done)
  5. Also fix bug in Qdrant - state and recent schemas are incorrectly retrieved (somehow sort order was ignored during implementation) (90% fixed)
  6. Several tests must be ported (some unit, state sync - done, sql job client)

notes:
sql job client must know how identifiers in information schema are stored:

  • redshift always lower case
  • we force upper case in snowflake, perhaps with the fixes above we can do both case sensitive and case insensitive options
  • other destinations seem to store tables as quoted in CREATE and ALTER statements

(4) an overview of case sensitivity and preferred naming convention per destination: https://www.linkedin.com/posts/toby-mao_sql-activity-7170104665412423680-17gv/

Copy link

netlify bot commented Feb 25, 2024

Deploy Preview for dlt-hub-docs ready!

Name Link
🔨 Latest commit 2a7c5dd
🔍 Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/65f843e5046ca7000847dd18
😎 Deploy Preview https://deploy-preview-998--dlt-hub-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@rudolfix rudolfix self-assigned this Mar 4, 2024
@rudolfix rudolfix added the community This issue came from slack community workspace label Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community This issue came from slack community workspace
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

None yet

1 participant