Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): Defer column file opening on read #3664

Closed
ideoma opened this issue Aug 18, 2023 · 0 comments · Fixed by #4500
Closed

feat(core): Defer column file opening on read #3664

ideoma opened this issue Aug 18, 2023 · 0 comments · Fixed by #4500
Labels
Core Related to storage, data type, etc. Enhancement Enhance existing functionality

Comments

@ideoma
Copy link
Collaborator

ideoma commented Aug 18, 2023

Describe the bug

With hourly partitions, the number of column files grows quickly. A lot of syscalls make to open and map all the files on TableReader partition open resulting in log lines

TableReader open partition table/2022-11-07T11.7176 [rowCount=1335278, partitionNameTxn=7176, transientRowCount=1912134, partitionIndex=30, partitionCount=6159]

Not all the columns might be read by the query and it can save a significant number of syscalls (and query time) to not open the files.

To reproduce

No response

Expected Behavior

No response

Environment

- **QuestDB version**:7.3.1

Additional context

No response

@puzpuzpuz puzpuzpuz added Enhancement Enhance existing functionality Core Related to storage, data type, etc. labels May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Core Related to storage, data type, etc. Enhancement Enhance existing functionality
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants