Git LFS doesn't appear to be working

[stonesbg]

🔧 Summary

We have lfs setup on our environments and after installing the lefthook hooks it replaces the git lfs hooks. From the documentation it lefthook does support lfs and if I turn on verbose I can see it try to execute lfs but not sure if there is secret sauce in the lfs hooks but every push doens't include the lfs file upload.

I have tried both wsl2 and windows and both behave the same. If I remove the lefthook pre-push hooks and re-instate the lfs ones using git lfs update --force. If i push then i can see that the lfs update has triggered.

The only thing I can think of is that potentially would be different execution environment between the two. I noticed that the hook uses git-lfs and the lefthook call uses git lfs. Lastly I also noticed in the hook for windows it does a unset GIT_PREFIX; ssh git@bitbucket.org 'git-receive-pack '\''brian_garvey/lfs-lefthook-test.git'\'''

Here is the test repo LFS Lefthook Test. I tried to keep it simple.

Lefthook version

lefthook -> 1.4.3
git -> 2.41.0
git-lfs -> 3.0.2 (WSL 2) & 3.3.0 (windows)

Steps to reproduce

1. Copy JPG File to directory
2. Install the lefthook hooks lefthook install -a
3. Run git add . and git commit -m 'message'
4. Rung git push

Expected results

LFS images are pushed to the remote repository and and subsequently pull them down

Actual results

Check bitbucket UI and see that images are not loading and external pull shows 404 errors for the lfs files

Logs / Screenshots

git hook version

trace git-lfs: pure SSH protocol connection failed: Unable to negotiate version with remote side (unable to read capabilities): EOF
trace git-lfs: pre-push: refs/heads/master a92e62630365e8950c18a69b83ab8c141f97a4d8 refs/heads/master eee5ea60a973bde0729800020680bfd928d6be24
trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '--tags' '-q' 'origin'
trace git-lfs: tq: running as batched queue, batch size of 100

lefthook version

trace git-lfs: pure SSH protocol connection failed: Unable to negotiate version with remote side (unable to read capabilities): EOF
trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '--tags' '-q' 'origin'
trace git-lfs: filepathfilter: creating pattern ".git" of type gitignore

[mrexox]

Hey! Thank you for the issue. There were always difficulties with LFS because lefthook replaces the hook generated by LFS.

Could you send the output of the LEFTHOOK_VERBOSE=true git push?

[stonesbg]

Here you go. I was wondering because there is no STDIN available in lefthook maybe it is appending empty lfs refs and it thinks it has done everything it needs.

❯ git push
Lefthook v1.4.1
RUNNING HOOK: pre-push
[git-lfs] executing hook: git lfs pre-push origin git@bitbucket.org:brian_garvey/lfs-lefthook-test.git
[lefthook] cmd: [git diff --name-only HEAD @{push}]
[lefthook] err: <nil>
[lefthook] out: PH03425I.JPG

[lefthook] files before filters:
[PH03425I.JPG]
[lefthook] files after filters:
[PH03425I.JPG]
[lefthook] files after escaping:
[PH03425I.JPG]
[lefthook] executing: echo 'LFS running'

  EXECUTE > lfs
LFS running


SUMMARY: (done in 11.28 seconds)
✔️  lfs
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 393 bytes | 65.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
To bitbucket.org:brian_garvey/lfs-lefthook-test.git
   02c86d2..2dbbeb7  master -> master

[avudnez]

We're having the same error sadly, since using lefthook the lfs objects do not get pushed and we encounter smudge errors when trying to fetch the commits containing them.
The environment is Git Bash on Windows, with lefthook installed from winget.

Here's an example of an usual push that should push lfs objects but doesnt:

$ LEFTHOOK_VERBOSE=1 git push origin the_branch:the_branch
+ '[' '' = 0 ']'
+ call_lefthook run pre-push origin https://remote
++ git rev-parse --show-toplevel
+ dir=C:/DNE/Git/monorepo-dev
++ uname
++ tr '[:upper:]' '[:lower:]'
+ osArch=mingw64_nt-10.0-19045
++ uname -m
++ sed s/aarch64/arm64/
+ cpuArch=x86_64
+ test -n ''
+ lefthook.exe -h
+ lefthook.bat -h
+ test -f C:/DNE/Git/monorepo-dev/node_modules/lefthook/bin/index.js
+ test -f C:/DNE/Git/monorepo-dev/node_modules/@evilmartians/lefthook/bin/lefthook_mingw64_nt-10.0-19045_x86_64/lefthook.exe
+ test -f C:/DNE/Git/monorepo-dev/node_modules/@evilmartians/lefthook-installer/bin/lefthook_mingw64_nt-10.0-19045_x86_64/lefthook.exe
+ bundle exec lefthook -h
+ yarn lefthook -h
+ pnpm lefthook -h
+ swift package plugin lefthook
+ command -v npx
+ npx lefthook run pre-push origin https://remote
│ [lefthook] cmd: [git rev-parse --show-toplevel]
│ [lefthook] dir:
│ [lefthook] err: <nil>
│ [lefthook] out: C:/DNE/Git/monorepo-dev
│ [lefthook] cmd: [git rev-parse --git-path hooks]
│ [lefthook] dir:
│ [lefthook] err: <nil>
│ [lefthook] out: .git/hooks
│ [lefthook] cmd: [git rev-parse --git-path info]
│ [lefthook] dir:
│ [lefthook] err: <nil>
│ [lefthook] out: .git/info
│ [lefthook] cmd: [git rev-parse --git-dir]
│ [lefthook] dir:
│ [lefthook] err: <nil>
│ [lefthook] out: .git
│ [lefthook] cmd: [git hash-object -t tree /dev/null]
│ [lefthook] dir:
│ [lefthook] err: <nil>
│ [lefthook] out: 4b825dc642cb6eb9a060e54bf8d69288fbee4904
│ [git-lfs] executing hook: git lfs pre-push origin https://remote
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
remote:
remote: Create a new pull request for 'the_branch':
remote:   https://remote
remote:
remote: . Processing 1 references
remote: Processed 1 references in total
To https://remote
 * [new branch]            the_branch -> the_branch
 *

git lfs seems to be called, but the objects are not found on the remote.

Trying to get more information with GIT_TRACE=1 leads to an error in git:

$ LEFTHOOK_VERBOSE=1 GIT_TRACE=1 git push origin the_branch:the_branch
11:00:09.850967 exec-cmd.c:244          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
11:00:09.857969 git.c:463               trace: built-in: git push origin the_branch:the_branch
11:00:09.858969 run-command.c:659       trace: run_command: GIT_DIR=.git git remote-https origin https://remote
11:00:09.868974 exec-cmd.c:244          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:00:09.874980 git.c:749               trace: exec: git-remote-https origin https://remote
11:00:09.874980 run-command.c:659       trace: run_command: git-remote-https origin https://remote
11:00:09.886982 exec-cmd.c:244          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:00:10.468957 run-command.c:659       trace: run_command: 'git credential-manager get'
11:00:10.987143 exec-cmd.c:244          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:00:10.993148 git.c:749               trace: exec: git-credential-manager get
11:00:10.993148 run-command.c:659       trace: run_command: git-credential-manager get
11:00:11.056920 exec-cmd.c:244          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:00:11.063584 git.c:463               trace: built-in: git config --null --list
11:00:12.013188 run-command.c:659       trace: run_command: 'git credential-manager store'
11:00:12.054024 exec-cmd.c:244          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:00:12.060026 git.c:749               trace: exec: git-credential-manager store
11:00:12.060026 run-command.c:659       trace: run_command: git-credential-manager store
11:00:12.126649 exec-cmd.c:244          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:00:12.133892 git.c:463               trace: built-in: git config --null --list
11:00:12.345316 run-command.c:659       trace: run_command: .git/hooks/pre-push origin https://remote
+ '[' '' = 0 ']'
+ call_lefthook run pre-push origin https://remote
++ git rev-parse --show-toplevel
11:00:12.398168 exec-cmd.c:244          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:00:12.404169 git.c:463               trace: built-in: git rev-parse --show-toplevel
+ dir=C:/DNE/Git/monorepo-dev
++ uname
++ tr '[:upper:]' '[:lower:]'
+ osArch=mingw64_nt-10.0-19045
++ uname -m
++ sed s/aarch64/arm64/
+ cpuArch=x86_64
+ test -n ''
+ lefthook.exe -h
+ lefthook.bat -h
+ test -f C:/DNE/Git/monorepo-dev/node_modules/lefthook/bin/index.js
+ test -f C:/DNE/Git/monorepo-dev/node_modules/@evilmartians/lefthook/bin/lefthook_mingw64_nt-10.0-19045_x86_64/lefthook.exe
+ test -f C:/DNE/Git/monorepo-dev/node_modules/@evilmartians/lefthook-installer/bin/lefthook_mingw64_nt-10.0-19045_x86_64/lefthook.exe
+ bundle exec lefthook -h
+ yarn lefthook -h
+ pnpm lefthook -h
+ swift package plugin lefthook
+ command -v npx
+ npx lefthook run pre-push origin https://remote
│ [lefthook] cmd: [git rev-parse --show-toplevel]
│ [lefthook] dir:
│ [lefthook] err: <nil>
│ [lefthook] out: 11:00:15.987374 exec-cmd.c:244          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:00:15.992376 git.c:463               trace: built-in: git rev-parse --show-toplevel
C:/DNE/Git/monorepo-dev
│ [lefthook] cmd: [git rev-parse --git-path hooks]
│ [lefthook] dir:
│ [lefthook] err: <nil>
│ [lefthook] out: 11:00:16.022296 exec-cmd.c:244          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:00:16.029019 git.c:463               trace: built-in: git rev-parse --git-path hooks
.git/hooks
│ [lefthook] cmd: [git rev-parse --git-path info]
│ [lefthook] dir:
│ [lefthook] err: <nil>
│ [lefthook] out: 11:00:16.064186 exec-cmd.c:244          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:00:16.069310 git.c:463               trace: built-in: git rev-parse --git-path info
.git/info
Error: mkdir .\11:00:16.064186 exec-cmd.c:244          trace: resolved executable dir: C:\Program Files\Git\mingw64\libexec\git-core
11:00:16.069310 git.c:463               trace: built-in: git rev-parse --git-path info
.git\info: The filename, directory name, or volume label syntax is incorrect.
error: failed to push some refs to 'https://remote'

Running git lfs push origin the_branch manually solves the issue.
Unfortunately this is blocking adoption in our workflow.

[mrexox]

Hey! Actually lefthook is just calling what Git LFS adds in the .git/hooks/pre-push with you run git lfs update --force. The content I get when running this in the pre-push hook is the following:

#!/bin/sh
command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting the 'pre-push' file in the hooks directory (set by 'core.hookspath'; usually '.git/hooks').\n"; exit 2; }
git lfs pre-push "$@"

So, it calls git lfs pre-push with args of the hook. That's exactly what lefthook does.

Could you show what's in your hook after running git lfs update --force? Does Git LFS work with LFS-native hooks, without lefthook?

[avudnez]

Thanks for the quick response.

Could you show what's in your hook after running git lfs update --force?

Same as you, the usual git lfs hook.

Does Git LFS work with LFS-native hooks, without lefthook?

Yes it does, we confirmed it after a lefthook uninstall.

[mrexox]

Does Git LFS work with LFS-native hooks, without lefthook?

Yes it does, we confirmed it after a lefthook uninstall.

So, it calls the same git lfs pre-push origin https://remote but succeedes? 🤔

[avudnez]

So, it calls the same git lfs pre-push origin https://remote but succeedes? 🤔

Yes, and as far as I can tell pushing with lefthook installed succeeds, but the objects don't end up on the server which is strange.
Sadly I can't check what's going on when enabling GIT_TRACE=1 because it makes git error out.

[mrexox]

That's weird. I will try to investigate, but please come back with any information you discover.

[phisko]

Hi, have you had the chance to take a look at this? Do you have an ETA (even very approximate) on when you might? We've recently migrated from pre-commit to lefthook, but might have to migrate back if there's no fix on the horizon

(I hope this doesn't sound passive-aggressive, I'm just trying to get a clear picture so we can decide what to do on our end)

[tdesveaux]

@mrexox
Hi, I looked a bit and I believe the issue is that git lfs pre-push expect to read commit information on STDIN.
This is documented, here is the first few lines of git lfs pre-push --help:

git lfs pre-push <remote> [remoteurl]

Responds to Git pre-hook events. It reads the range of commits from
STDIN, in the following format:


<local-ref> SP <local-sha1> SP <remote-ref> SP <remote-sha1> \n

This issue has an easy reproduction in a repository with pre-push hooks.
After a push of a new/updated LFS file, simply run: rm -rf .git/lfs/ && git lfs fetch origin HEAD

[tdesveaux]

Tried adding a dumb cmd.Stdin = os.Stdin in execute_unix.go RawExecute and it works now.
I believe it's not a proper solution as if another hook needs the information from stdin, execution order will let the first hook work as expected, and others fail.

Lefthook should probably read stdin and provide it to all hooks as needed.

[mrexox]

@tdesveaux , it makes sense to me. Thank you for checking the RawExecute method. I agree that all hooks should receive the same STDIN in case of pre-push hook.

I will try to think out this feature. The only complication here is that some commands may be executed in interactive mode, so we can't just read STDIN into a buffer for them. But I guess this is anyway feasible to find a workaround.

[tdesveaux]

If I can suggest a quick "fix" for the moment.
Since you already have the use_stdin option for commands, you could throw an error if a pre-push command has it to true when LFS is also detected?

This way, you can add cmd.Stdin = os.Stdin to RawExecute and fix LFS push.
The feature to have multiple commands that need stdin can be implemented later on.

I'm not knowledgeable on the lefthook codebase at all, so this might not be possible.

If this suggestion is fine with you, I could try to implement it.

[mrexox]

I like your suggestion. I'd be glad to receive a PR with this fix 👍