Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add command tsort #2984

Merged
merged 1 commit into from
May 27, 2024
Merged

Add command tsort #2984

merged 1 commit into from
May 27, 2024

Conversation

jbduncan
Copy link
Contributor

@jbduncan jbduncan commented May 6, 2024

Add cmds/exp/tsort, which aims to implement POSIX tsort (except for "Environment Variables").

Currently I'm undecided what to do about cycles, as handling them is tricky and POSIX takes no stance on them.

There are still some remaining TODOs in the source code.

Resolves #2982.

@rminnich rminnich added the Awaiting reviewer Waiting for a reviewer. label May 7, 2024
Copy link
Member

@rminnich rminnich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you need to vendor all that code, now that we use modules?

cmds/exp/tsort/tsort.go Outdated Show resolved Hide resolved
@jbduncan
Copy link
Contributor Author

jbduncan commented May 7, 2024

Did you need to vendor all that code, now that we use modules?

@rminnich Good question! I assumed that all dependencies were vendored in this project. Am I mistaken?

Anyway, my test needs the cmpopts package from github.com/google/go-cmp, and the build fails if I don't vendor it, so I was wondering what's the recommended way of importing it?

@rminnich
Copy link
Member

rminnich commented May 7, 2024

did you do a go mod update? That usually does it for me?

@jbduncan
Copy link
Contributor Author

jbduncan commented May 8, 2024

did you do a go mod update? That usually does it for me?

I did, and apparently it doesn't work.

After reverting the vendor update with 48710d2, running go get -u -t ./... && go mod tidy && go mod download, and finally running go build ./..., this is what I get.

This suggests to me that vendoring is mandatory. Furthermore, my IDE doesn't see the github.com/google/go-cmp/cmp/cmpopts package.

Am I missing something obvious?

go: inconsistent vendoring in /Users/jonathan.bluett-dunc/dev/u-root:
        github.com/ProtonMail/go-crypto@v1.0.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/beevik/ntp@v1.4.1: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/cenkalti/backoff/v4@v4.3.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/gliderlabs/ssh@v0.3.7: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/gojuno/minimock/v3@v3.3.7: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/google/go-cmp@v0.6.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/google/go-tpm@v0.9.1-0.20240411180339-1fb84445f623: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/google/uuid@v1.6.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/hugelgupf/vmtest@v0.0.0-20240307030256-5d9f3d34a58d: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/insomniacslk/dhcp@v0.0.0-20240419123447-f1cffa2c0c49: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/intel-go/cpuid@v0.0.0-20220614022739-219e067757cb: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/kevinburke/ssh_config@v1.2.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/klauspost/compress@v1.17.8: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/pierrec/lz4/v4@v4.1.21: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/rekby/gpt@v0.0.0-20200614112001-7da10aec5566: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/safchain/ethtool@v0.3.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/u-root/gobusybox/src@v0.0.0-20240305154353-d8fbaca23e26: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/u-root/mkuimage@v0.0.0-20240228022452-899a47eaaa31: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/u-root/uio@v0.0.0-20240224005618-d2acac8f3701: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/ulikunitz/xz@v0.5.12: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        golang.org/x/crypto@v0.23.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        golang.org/x/exp@v0.0.0-20240506185415-9bf2ced13842: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        golang.org/x/net@v0.25.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        golang.org/x/sys@v0.20.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        golang.org/x/term@v0.20.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        golang.org/x/text@v0.15.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        golang.org/x/tools@v0.21.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        gopkg.in/yaml.v2@v2.4.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        mvdan.cc/sh/v3@v3.8.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        src.elv.sh@v0.20.1: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/charmbracelet/bubbles@v0.18.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/charmbracelet/bubbletea@v0.26.1: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/charmbracelet/lipgloss@v0.10.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/cloudflare/circl@v1.3.8: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/erikgeiser/coninput@v0.0.0-20211004153227-1c3628e74d0f: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/jsimonetti/rtnetlink@v1.4.1: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/mattn/go-runewidth@v0.0.15: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/mdlayher/socket@v0.5.1: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/muesli/termenv@v0.15.2: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/rivo/uniseg@v0.4.7: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/sahilm/fuzzy@v0.1.1: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        golang.org/x/mod@v0.17.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        golang.org/x/sync@v0.7.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        gopkg.in/yaml.v3@v3.0.1: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/ProtonMail/go-crypto@v0.0.0-20221026131551-cf6655e29de4: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/beevik/ntp@v0.3.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/cenkalti/backoff/v4@v4.1.3: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/charmbracelet/bubbles@v0.15.1-0.20230123181021-a6a12c4a31eb: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/charmbracelet/bubbletea@v0.24.1: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/charmbracelet/lipgloss@v0.7.1: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/cloudflare/circl@v1.3.7: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/containerd/console@v1.0.4-0.20230706203907-8f6c4e4faef5: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/gliderlabs/ssh@v0.1.2-0.20181113160402-cbabf5414432: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/gojuno/minimock/v3@v3.0.8: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/google/go-cmp@v0.5.9: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/google/go-tpm@v0.9.1-0.20230914180155-ee6cbcd136f8: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/google/uuid@v1.3.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/hugelgupf/vmtest@v0.0.0-20240228002643-de15f4612e10: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/insomniacslk/dhcp@v0.0.0-20231206064809-8c70d406f6d2: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/intel-go/cpuid@v0.0.0-20200819041909-2aa72927c3e2: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/jsimonetti/rtnetlink@v1.3.5: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/kevinburke/ssh_config@v1.1.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/klauspost/compress@v1.17.4: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/mattn/go-runewidth@v0.0.14: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/mdlayher/socket@v0.5.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/muesli/termenv@v0.15.1: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/pierrec/lz4/v4@v4.1.14: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/rekby/gpt@v0.0.0-20200219180433-a930afbc6edc: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/rivo/uniseg@v0.4.4: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/safchain/ethtool@v0.0.0-20200218184317-f459e2d13664: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/sahilm/fuzzy@v0.1.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/u-root/gobusybox/src@v0.0.0-20240225013946-a274a8d5d83a: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/u-root/mkuimage@v0.0.0-20240225063926-11a3bcc79c2a: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/u-root/uio@v0.0.0-20240209044354-b3d14b93376a: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/ulikunitz/xz@v0.5.11: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/crypto@v0.21.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/exp@v0.0.0-20240222234643-814bf88cf225: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/mod@v0.15.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/net@v0.23.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/sync@v0.6.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/sys@v0.18.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/term@v0.18.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/text@v0.14.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/tools@v0.18.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        gopkg.in/yaml.v2@v2.2.8: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        gopkg.in/yaml.v3@v3.0.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        mvdan.cc/sh/v3@v3.7.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        src.elv.sh@v0.16.0-rc1.0.20220116211855-fda62502ad7f: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod

        To ignore the vendor directory, use -mod=readonly or -mod=mod.
        To sync the vendor directory, run:
                go mod vendor

Copy link

codecov bot commented May 8, 2024

Codecov Report

Attention: Patch coverage is 96.79144% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 56.03%. Comparing base (752942d) to head (260ccb9).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2984      +/-   ##
==========================================
+ Coverage   55.61%   56.03%   +0.42%     
==========================================
  Files         512      517       +5     
  Lines       31139    31326     +187     
==========================================
+ Hits        17319    17555     +236     
+ Misses      13820    13771      -49     
Flag Coverage Δ
.-amd64 90.69% <ø> (ø)
cmds/...-amd64 45.49% <96.25%> (+0.76%) ⬆️
integration/generic-tests/...-amd64 19.80% <ø> (ø)
integration/generic-tests/...-arm 11.69% <ø> (ø)
integration/generic-tests/...-arm64 23.96% <ø> (ø)
integration/gotests/...-amd64 61.00% <96.79%> (+0.75%) ⬆️
integration/gotests/...-arm 61.64% <96.79%> (+0.41%) ⬆️
integration/gotests/...-arm64 62.01% <96.79%> (+0.39%) ⬆️
pkg/...-amd64 56.00% <ø> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
everything 62.02% <ø> (+0.21%) ⬆️
cmds/exp 30.54% <96.79%> (+2.14%) ⬆️

Copy link
Member

@rminnich rminnich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

another question, sorry :-)

Can you make a tsort package and have it implement the interface:
https://medium.com/@kdnotes/sort-sort-interface-in-golang-1d263d96956d

@rminnich rminnich added Awaiting author Waiting for new changes or feedback for author. and removed Awaiting reviewer Waiting for a reviewer. labels May 10, 2024
@rminnich
Copy link
Member

also, I think my interface question was dumb, so nvm. I will test out your PR

cmds/exp/tsort/tsort.go Outdated Show resolved Hide resolved
@rminnich
Copy link
Member

I pulled your PR, removed the vendoring,
did a go mod tidy
and yeah, it wants to vendor, I'm always a bit mystified on what drives the rules on vendoring.
Thanks.

@jbduncan
Copy link
Contributor Author

also, I think my interface question was dumb, so nvm. I will test out your PR

@rminnich No worries, I totally understand. I made the same mistake too when first learning about tsort. 😁

@jbduncan jbduncan marked this pull request as ready for review May 12, 2024 16:56
@jbduncan
Copy link
Contributor Author

jbduncan commented May 12, 2024

I'm tempted to move the topologicalOrdering function into pkg, but I'm not fond of its callback style, and I'd rather wait until range-over-func is available in Go to migrate the callbacks to an iterator style. So I'll leave topologicalOrdering where it is for now.

@jbduncan
Copy link
Contributor Author

@rminnich @binjip978 I'm ready for another review when you are!

When we're done, I intend to make two signed-off commits, one for the github.com/google/go-cmp upgrade/vendoring and another for tsort itself.

@jbduncan
Copy link
Contributor Author

By the way, I noticed that CIFuzz didn't catch a bug I found towards the end of development, so I was wondering if there's a way to tell it to fuzz tsort directly, if it doesn't do that already?

@jbduncan
Copy link
Contributor Author

I've squashed many bugs over the last few days, so I'm doubly ready!

@binjip978 binjip978 added Awaiting reviewer Waiting for a reviewer. and removed Awaiting author Waiting for new changes or feedback for author. labels May 18, 2024
Copy link
Member

@rminnich rminnich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we are very close. Let's get this last cycle of reviews done and this can go in.

Thanks for your contribution.

@rminnich
Copy link
Member

Dont' forget to git commit --amend -s

@rminnich
Copy link
Member

Also, we like to maintain the rule that the tests work for any commit. Might be best to squash these many commits? Thanks.

@rminnich rminnich added Awaiting author Waiting for new changes or feedback for author. and removed Awaiting reviewer Waiting for a reviewer. labels May 18, 2024
@jbduncan
Copy link
Contributor Author

I think we are very close. Let's get this last cycle of reviews done and this can go in.

Thanks for your contribution.

You're very welcome, this was fun to work on!

Dont' forget to git commit --amend -s

Okay, on it. 👍

Also, we like to maintain the rule that the tests work for any commit. Might be best to squash these many commits? Thanks.

Cool, I was planning to squash and sign the commits sooner or later, I just wasn't sure when you wanted me to do so, so I'm on it.

@jbduncan jbduncan force-pushed the tsort branch 3 times, most recently from fb27180 to 5c42388 Compare May 19, 2024 10:58
@jbduncan
Copy link
Contributor Author

@rminnich @binjip978 I'm ready for another review whenever you are.

@jbduncan
Copy link
Contributor Author

I think the code coverage has gone down because, due to the prevalence of maps in this implementation, one or two of the code paths aren't always visited by the tests. I'm not entirely sure what to do about them, so I'm happy to leave it to your judgement.

This introduces a version of tsort that follows the POSIX tsort
specification except for the "Environment Variables" section:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/tsort.html

Signed-off-by: Jonathan Bluett-Duncan <jbluettduncan@gmail.com>
@jbduncan
Copy link
Contributor Author

jbduncan commented May 25, 2024

@rminnich @binjip978, I see that @binjip978 approved things a week ago, so I was wondering if there's anything else I need to do before this can be merged in?

I'm aware there was a drop in code coverage at some point, but I don't think it's an issue. I think it's just flaky because the implementation relies on maps heavily, so one or two of the code paths aren't always visited by the tests.

@binjip978 binjip978 added Awaiting reviewer Waiting for a reviewer. and removed Awaiting author Waiting for new changes or feedback for author. labels May 25, 2024
@binjip978
Copy link
Contributor

I think the code coverage has gone down because, due to the prevalence of maps in this implementation, one or two of the code paths aren't always visited by the tests. I'm not entirely sure what to do about them, so I'm happy to leave it to your judgement.

No worries, I updated the label, but if no new comments will follow I will merge it soon.

@binjip978 binjip978 merged commit d92d00a into u-root:main May 27, 2024
37 checks passed
@jbduncan jbduncan deleted the tsort branch May 27, 2024 11:38
@jbduncan
Copy link
Contributor Author

Woo! Really happy to see this merged in, so thank you for reviewing this and being so receptive, @binjip978 and @rminnich. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Awaiting reviewer Waiting for a reviewer.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

tsort
3 participants