feat: add `inspect report` subcommand and embed queries as SQL scripts #2246

encima · 2024-05-06T14:54:02Z

What kind of change does this PR introduce?

Chore: split queries out into embedded SQL scripts
Feature: add an inspect db report subcommand that runs all queries and saves their output to CSV files in the current directory

What is the current behavior?

Queries currently stored inside a go submodule

What is the new behavior?

Embedded SQL scripts and report subcommand for inspect that runs all queries and saves their output to the current directory as CSV files

Additional context

Add any other context or screenshots.

SQL scripts

coveralls · 2024-05-06T15:00:28Z

Pull Request Test Coverage Report for Build 8985859965

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

2 of 68 (2.94%) changed or added relevant lines in 21 files are covered.
5 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.3%) to 57.204%

Changes Missing Coverage	Changed/Added Lines	%
internal/inspect/bloat/bloat.go	1	0.0%
internal/inspect/blocking/blocking.go	1	0.0%
internal/inspect/calls/calls.go	1	0.0%
internal/inspect/index_sizes/index_sizes.go	1	0.0%
internal/inspect/index_usage/index_usage.go	1	0.0%
internal/inspect/locks/locks.go	1	0.0%
internal/inspect/long_running_queries/long_running_queries.go	1	0.0%
internal/inspect/outliers/outliers.go	1	0.0%
internal/inspect/replication_slots/replication_slots.go	1	0.0%
internal/inspect/role_connections/role_connections.go	1	0.0%

Files with Coverage Reduction	New Missed Lines	%
internal/gen/keys/keys.go	5	11.76%

Totals
Change from base Build 8971086947:	-0.3%
Covered Lines:	6392
Relevant Lines:	11174

💛 - Coveralls

KevinBrolly · 2024-05-07T09:14:35Z

LGTM but still needs review from CLI team 👍

sweatybridge

Adding a reports command is nice. Apart from those review comments, another suggestion is to wrap all queries in a transaction so they are measuring the same state of the db.

internal/inspect/utils.go

Co-authored-by: Han Qiao <sweatybridge@gmail.com>

encima · 2024-05-07T11:05:51Z

Adding a reports command is nice. Apart from those review comments, another suggestion is to wrap all queries in a transaction so they are measuring the same state of the db.

This is certainly possible, although the current state is not time critical in my mind. The report is useful for insights and debugging but not realtime insights. It could be useful to remove more time-sensitive queries (like locks) from the command

Bumps [tar](https://github.com/isaacs/node-tar) from 7.0.1 to 7.1.0. - [Release notes](https://github.com/isaacs/node-tar/releases) - [Changelog](https://github.com/isaacs/node-tar/blob/main/CHANGELOG.md) - [Commits](isaacs/node-tar@v7.0.1...v7.1.0) --- updated-dependencies: - dependency-name: tar dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…58.0 (#2241) chore(deps): bump github.com/golangci/golangci-lint Bumps [github.com/golangci/golangci-lint](https://github.com/golangci/golangci-lint) from 1.57.2 to 1.58.0. - [Release notes](https://github.com/golangci/golangci-lint/releases) - [Changelog](https://github.com/golangci/golangci-lint/blob/master/CHANGELOG.md) - [Commits](golangci/golangci-lint@v1.57.2...v1.58.0) --- updated-dependencies: - dependency-name: github.com/golangci/golangci-lint dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [golang.org/x/term](https://github.com/golang/term) from 0.19.0 to 0.20.0. - [Commits](golang/term@v0.19.0...v0.20.0) --- updated-dependencies: - dependency-name: golang.org/x/term dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.19.0 to 0.20.0. - [Commits](golang/oauth2@v0.19.0...v0.20.0) --- updated-dependencies: - dependency-name: golang.org/x/oauth2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github.com/gin-gonic/gin](https://github.com/gin-gonic/gin) from 1.9.1 to 1.10.0. - [Release notes](https://github.com/gin-gonic/gin/releases) - [Changelog](https://github.com/gin-gonic/gin/blob/master/CHANGELOG.md) - [Commits](gin-gonic/gin@v1.9.1...v1.10.0) --- updated-dependencies: - dependency-name: github.com/gin-gonic/gin dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 5 to 6. - [Release notes](https://github.com/golangci/golangci-lint-action/releases) - [Commits](golangci/golangci-lint-action@v5...v6) --- updated-dependencies: - dependency-name: golangci/golangci-lint-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: clean up db pull implementation * chore: update unit tests

…58.1 (#2256) chore(deps): bump github.com/golangci/golangci-lint Bumps [github.com/golangci/golangci-lint](https://github.com/golangci/golangci-lint) from 1.58.0 to 1.58.1. - [Release notes](https://github.com/golangci/golangci-lint/releases) - [Changelog](https://github.com/golangci/golangci-lint/blob/master/CHANGELOG.md) - [Commits](golangci/golangci-lint@v1.58.0...v1.58.1) --- updated-dependencies: - dependency-name: github.com/golangci/golangci-lint dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

….26.2 (#2257) chore(deps): bump github.com/charmbracelet/bubbletea Bumps [github.com/charmbracelet/bubbletea](https://github.com/charmbracelet/bubbletea) from 0.26.1 to 0.26.2. - [Release notes](https://github.com/charmbracelet/bubbletea/releases) - [Commits](charmbracelet/bubbletea@v0.26.1...v0.26.2) --- updated-dependencies: - dependency-name: github.com/charmbracelet/bubbletea dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…6.1.2+incompatible (#2261) chore(deps): bump github.com/docker/cli Bumps [github.com/docker/cli](https://github.com/docker/cli) from 26.1.1+incompatible to 26.1.2+incompatible. - [Commits](docker/cli@v26.1.1...v26.1.2) --- updated-dependencies: - dependency-name: github.com/docker/cli dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…2262) Bumps [github.com/containers/common](https://github.com/containers/common) from 0.58.2 to 0.58.3. - [Release notes](https://github.com/containers/common/releases) - [Commits](containers/common@v0.58.2...v0.58.3) --- updated-dependencies: - dependency-name: github.com/containers/common dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…o 26.1.2+incompatible (#2263) chore(deps): bump github.com/docker/docker Bumps [github.com/docker/docker](https://github.com/docker/docker) from 26.1.1+incompatible to 26.1.2+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v26.1.1...v26.1.2) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

addressed with tests

encima · 2024-05-16T15:12:28Z

Failing only on tests about table sizes and record counts, oddly.

encima added 3 commits May 5, 2024 21:58

initial work to split out queries into standalone scripts

d6f1396

add report command to export all command output to CSV files and embed

b22a81d

SQL scripts

add progress printing to report command

258b54a

encima requested a review from a team as a code owner May 6, 2024 14:54

add output flag to specify directory

14f3926

sweatybridge previously requested changes May 7, 2024

View reviewed changes

internal/inspect/utils.go Outdated Show resolved Hide resolved

internal/inspect/utils.go Outdated Show resolved Hide resolved

internal/inspect/utils.go Outdated Show resolved Hide resolved

internal/inspect/utils.go Outdated Show resolved Hide resolved

Update internal/inspect/utils.go

7bb3427

Co-authored-by: Han Qiao <sweatybridge@gmail.com>

use pgx for csv output and rename flag for output path

a77b7b8

encima force-pushed the feat/extract-sql branch from 95e5773 to a77b7b8 Compare May 7, 2024 13:06

dependabot bot and others added 18 commits May 13, 2024 14:44

fix: bump edge-runtime to 1.47.0

542fd37

fix: bump studio version (#2245)

93dc400

fix: bump edge-runtime to 1.48.0

076a244

chore: clean up db pull implementation (#2175)

93c28f0

* chore: clean up db pull implementation * chore: update unit tests

feat: add hook secrets

1d3b5f4

Apply suggestions from code review

6d8041f

fix: add send sms hook as test

fa06642

chore: typo in variable name

c098332

fix: index out of bound when applying migrations

4db76cc

chore: support struct return type for pgmock

420dbf6

sweatybridge and others added 7 commits May 13, 2024 14:45

fix: add unit test for cache hit

98cda14

fix: create migrations dir before saving diff (#2260)

6083413

fix: bump edge-runtime to 1.49.0

9e597e7

initial work to split out queries into standalone scripts

7dfd87f

encima force-pushed the feat/extract-sql branch from 5a66f9d to 0bce84c Compare May 14, 2024 07:29

add test cases for inspect commands

4b63d29

encima force-pushed the feat/extract-sql branch from 0bce84c to 4b63d29 Compare May 14, 2024 07:30

encima requested a review from sweatybridge May 14, 2024 07:32

Merge branch 'develop' into feat/extract-sql

bd592c1

encima force-pushed the feat/extract-sql branch from 0734f49 to bd592c1 Compare May 16, 2024 13:19

sweatybridge and others added 6 commits May 20, 2024 16:00

chore: use pg schemas in unit tests

153b65e

Merge branch 'develop' into feat/extract-sql

132d102

chore: test report command

21c7c81

chore: verify connection is closed

708605d

chore: embed queries within each command package

18f408b

chore: move query reading to file walker

27d3d6f

sweatybridge approved these changes May 21, 2024

View reviewed changes

sweatybridge changed the title ~~Add inspect db report subcommand and embed queries as SQL scripts~~ feat: add inspect report subcommand and embed queries as SQL scripts May 21, 2024

sweatybridge merged commit ee621c5 into develop May 21, 2024
12 checks passed

sweatybridge deleted the feat/extract-sql branch May 21, 2024 12:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add `inspect report` subcommand and embed queries as SQL scripts #2246

feat: add `inspect report` subcommand and embed queries as SQL scripts #2246

encima commented May 6, 2024

coveralls commented May 6, 2024 •

edited

KevinBrolly commented May 7, 2024

sweatybridge left a comment

encima commented May 7, 2024

encima commented May 16, 2024

feat: add inspect report subcommand and embed queries as SQL scripts #2246

feat: add inspect report subcommand and embed queries as SQL scripts #2246

Conversation

encima commented May 6, 2024

What kind of change does this PR introduce?

What is the current behavior?

What is the new behavior?

Additional context

coveralls commented May 6, 2024 • edited

Pull Request Test Coverage Report for Build 8985859965

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

KevinBrolly commented May 7, 2024

sweatybridge left a comment

Choose a reason for hiding this comment

encima commented May 7, 2024

encima commented May 16, 2024

feat: add `inspect report` subcommand and embed queries as SQL scripts #2246

feat: add `inspect report` subcommand and embed queries as SQL scripts #2246

coveralls commented May 6, 2024 •

edited