Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reflection fails on complex nested types #269

Open
hsheth2 opened this issue Oct 27, 2023 · 0 comments
Open

Reflection fails on complex nested types #269

hsheth2 opened this issue Oct 27, 2023 · 0 comments

Comments

@hsheth2
Copy link

hsheth2 commented Oct 27, 2023

Describe the bug

For a table DDL like this:

CREATE TABLE IF NOT EXISTS bugtable on cluster 'local' (
	id Int,
	metadata Map(String, Map(String, Nullable(String)))
) ENGINE = MergeTree()
order by id

Reflection using get_columns() fails with Map.__init__() missing 1 required positional argument: 'value_type'

It looks like the issue is in this code

elif spec.startswith('Map'):
inner = spec[4:-1]
coltype = self.ischema_names['_map']
inner_types = [
self._get_column_type(name, t.strip())
for t in inner.split(',')
]
- the .split(',') is getting an input of String, Map(String, Nullable(String)) and splits it to [String, 'Map(String', 'Nullable(String))'] - it doesn't respect the parentheses. It could probably be replaced with a .split(',', maxsplit=1) to fix this issue (unless the key type was also complex, but that seems less likely)

Expected behavior
This should get reflected properly.

Originally discovered in the DataHub slack community: https://datahubspace.slack.com/archives/CUMUWQU66/p1698241313514559?thread_ts=1698112545.709559&cid=CUMUWQU66

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant