Long paths under windows dont work

[x29a]

I am facing https://sourceforge.net/p/cloc/bugs/101/ under Windows 7 SP1, 64bit with the bundled EXE version 1.62 and also 1.66 on NTFS filesystem.

Windows has a limit on the length of filenames. Perls Find::Find inherently has to deal with the limitation and therefor also cloc.

I am running cloc on a huge codebase, unfortunately, i cant create a minimal example presenting the problem or disclose the project structure. Running cloc on just the directory structure which poses the problem works fine, so i assume it has something todo when recursing through lots of folders.

Cloc is invoked with

--by-file --ignored ignored.log --found found.log --xml --out src.xml <relative path>

The error i get is:
Can't cd to ../../../.. from <relative path with length of 222>: No such file or directory at C:/StrawberryPerl/perl/lib/File/Find.pm line 469.

Note, i am running the .pl against Strawberry Perl 5.22.1.2-32bit instead of the .exe for debugging reasons .

The relative path is prepended by a path of length 35, resulting in a total length of 256 for the directory. The error results in not output file, therefor missing results.

One temporary fix is to subst the first part of the path to a new drive letter and hence save 33 chars.

[MichaelDimmitt]

@x29a ... I have a pull request contribution pending and no other contributions.
but I see two solutions to your problem.
the easiest way is a small change to the commit I have pending.
(1) find out how many sub folders you need to go.
(2)nest this command: for d in .// ; do (cd "$d" && !!); done
(3) putting the whole command inside of the explanation marks for however many levels till reach the parent folder's level.
(4) then cd into the final directory.
(5) and perform the same cloc function as normal.
example 3 subdirectories deep:
for d in .// ; do (cd "$d" && for d in ./*/ ; do (cd DirectoryContainingFile && cloc $(git ls-files)); done); done

or, second solution:
tokenize your folder path ... retaining a count of your tokens.
cd token1; cd token2; cd token3;cd DirectoryContainingFile; normal cloc command;
cd ..;#(count of numb tokens)

the use of semicolins is the reason these solutions exist for your issue. The solutions are splitting your large command into many small commands using the semicolin. Windows does less work per command and you should not have further problems.
Hope this helps. Cheers!

[AlDanial]

OK, shame on me for leaving this open nearly a year. The main issue is that I don't have ready access to a Windows machine--these days I resort to using an Amazon Web Services Windows instance, a bit of a hassle--so that diminishes my enthusiasm on tacking this.n

I'm now thinking I should be able to duplicate this on Linux, via some sym link trickery. I believe the ext4 file system, what I'm currently using, has a 255 limit on paths. I can make a legal <255 path with, for example, /tmp/aa/ab/ac/ .. then exceed the 255 by symlinking one of the middle directories to something with a longer name. I'll pursue that, and if no progress there I'll actually use a Windows machine.

[baffles]

Have there been any developments on this? Would using https://metacpan.org/pod/Win32::LongPath help to work around the issue on Windows?

[AlDanial]

There hasn't been progress. However, it is easier for me to get access to Windows now that it was in the past so there's hope. I looked at Win32::LongPath and wasn't able to see a long enhancement to the find() function which is where things currently go south.

[AlDanial]

I spent the past two hours researching and trying different techniques to run the File::Find module's find() function on Windows in a folder with > 260 characters in the path. No luck. I ruled out techniques that cd into the lower directories because this messes with downstream logic in cloc and is the reason I can't accept PR #59.

However, that technique may be the only way to get Perl on Windows to get a file listing. Here's my offer: if anyone can devise a Perl subroutine which accepts a Windows directory name as input and returns as output a list (or hash) of file paths (leading subdirectory name + file name) relative to the input directory, then I'll integrate this into cloc.

[x29a]

Not working with cloc under windows currently but i came across a windows filesystem specific workaround. By using the UNC prefix, the path limit might be circumvented. Can anybody give this a try?

The question would be if cloc should detect the filesystem and use the workaround or if users should just pass in the UNC'd path when facing problems.

[AlDanial]

cloc already detects the OS and does different path munging depending on Windows or Unix-like. I'm open to adding the UNC prefix for Windows in if anyone can vouch that it works through cloc.

[dluminello]

Hi All,
I was able to overcome the limitation from Perl by just referencing the path using "\\<SERVER_NAME><SHARE><FOLDER_NAME_PATH>". I had to "share" the folder first and give read permission.
For example
"\\fcoh4x-serv1\Test-CLOC\workspace\temp"

[RyanMcC]

I was able to work around File::Find on Windows by using a separate script to generate a list-file. Alternatively using --no-recurse should also work I think. But cloc still isn't able to read any files past the path length limit since it's using the native open/new IO::File and -X calls.

@AlDanial Would you be open to a PR using Win32::LongPath for these when running on Windows (and also maybe behind a flag), so that at least when using list-files or --no-recurse, cloc can work on files past the path length limit?

[AlDanial]

@RyanMcC I welcome the PR you describe!

[tfadler]

Hello any update on this? Thought I would ask. Thank you.

[x29a]

@tfadler since the issue is quite old, could you provide more information on what OS with which version and under which conditions you are facing the issue? exactly like i posted back in 2016?

@AlDanial is #736 maybe related?

[tfadler]

Great question. Sorry. I am on Windows 10

Trying to do diff

cloc --git --diff <commit id1> <commit id2>

and it did not work. I saw errors like "Can't cd to xxx". Unfortunately I already uninstalled and cleaned up things.

Our repo has very long filenames/depth.

I googled and got to this issue and it seemed similar/same to what I was seeing.

I was going to try doing the diff on linux later today hoping to work around the issue.

[AlDanial]

@tfadler : give the 1.96.1 Windows executable at https://github.com/AlDanial/cloc/releases a try. It includes PR #686 by @RyanMcC that uses Win32::LongPath to resolve long paths on Windows.

[tfadler]

Thank you @AlDanial
I should have provided the version I was using.

I got 1.96.1 from scoop. Let me know if you think that is stale. I checked and that is the reported version.

Note I tried and my commands are fine on Linux. Just thought I would mention that.

I looked the documentation and I did not see any param I needed to provide for long path but please correct me if I am wrong.

Here is a section of the output.

tar -x -f "C:\Users\USER1~1\AppData\Local\Temp\0HpvTnb5wy.tar" tar -x -f "C:\Users\USER1~1\AppData\Local\Temp\0HpvTnb5wy.tar_extra_00000001" tar -x -f "C:\Users\USER1~1\AppData\Local\Temp\0HpvTnb5wy.tar_extra_00000002" tar -x -f "C:\Users\USER1~1\AppData\Local\Temp\0HpvTnb5wy.tar_extra_00000003" tar -x -f "C:\Users\USER1~1\AppData\Local\Temp\0HpvTnb5wy.tar_extra_00000004" tar -x -f "C:\Users\USER1~1\AppData\Local\Temp\0HpvTnb5wy.tar_extra_00000005" tar -x -f "C:\Users\USER1~1\AppData\Local\Temp\0HpvTnb5wy.tar_extra_00000006" tar -x -f "C:\Users\USER1~1\AppData\Local\Temp\0HpvTnb5wy.tar_extra_00000007" tar -x -f "C:\Users\USER1~1\AppData\Local\Temp\0HpvTnb5wy.tar_extra_00000008" tar -x -f "C:\Users\USER1~1\AppData\Local\Temp\0HpvTnb5wy.tar_extra_00000009" tar -x -f "C:\Users\USER1~1\AppData\Local\Temp\0HpvTnb5wy.tar_extra_00000010" tar -x -f "C:\Users\USER1~1\AppData\Local\Temp\0HpvTnb5wy.tar_extra_00000011" tar -x -f "C:\Users\USER1~1\AppData\Local\Temp\0HpvTnb5wy.tar_extra_00000012" tar -x -f "C:\Users\USER1~1\AppData\Local\Temp\0HpvTnb5wy.tar_extra_00000013" tar -c -f "C:\Users\USER1~1\AppData\Local\Temp\0HpvTnb5wy.tar.final.tar" "C:\Users\USER1~1\AppData\Local\Temp\vCNd66OfRo" tar: Removing leading drive letter from member names mkdir C:\Users\USER1~1\AppData\Local\Temp\hmXCsuhngX cd C:\Users\USER1~1\AppData\Local\Temp\hmXCsuhngX tar -xf "C:\Users\USER1~1\AppData\Local\Temp\L3u8D__7lD.tar.final.tar" mkdir C:\Users\USER1~1\AppData\Local\Temp\wCWFLtiyje cd C:\Users\USER1~1\AppData\Local\Temp\wCWFLtiyje tar -xf "C:\Users\USER1~1\AppData\Local\Temp\0HpvTnb5wy.tar.final.tar" Using temp file list [C:\Users\USER1~1\AppData\Local\Temp\dd6xqrlFLo] Can't cd to ../../../.. from C:\Users\USER1~1\AppData\Local\Temp\hmXCsuhngX/Users/USER1~1/AppData/Local/Temp/6lfF2ZQA0J/web/Gui/src/app/xxxxxx-main-area/xxxxx-xxxxxxx-content-area/xxxxx-xxxxxxx-grid-area/xxxxx-xxxxxxx-module-information/xxxxx-xxxxxxx-module-information-event-grid: No such file or directory at <embedded>/File/Find.pm line 470. cannot remove path when cwd is C:/Users/USER1~1/AppData/Local/Temp/hmXCsuhngX/Users/USER1~1/AppData/Local/Temp/6lfF2ZQA0J/web/Gui/src/app/xxxxxx-main-area/xxxxx-xxxxxxx-content-area/xxxxx-xxxxxxx-grid-area/xxxxx-xxxxxxx-module-information/xxxxx-xxxxxxx-module-information-event-grid for C:/Users/USER1~1/AppData/Local/Temp/hmXCsuhngX: at <embedded>/File/Temp.pm line 921.

I had to obfuscate some but kept the file length the same. There are more similar lines after that.

I am not blocked since I can run on linux too.

I am running this in PowerShell 7.3.6. If that helps.

Let me know any thoughts. Thank you.

[taciano-perez]

I've just tried cloc 1.98 on Windows 10 and faced the same error. We also have a large codebase with very deep folder structure.

[AlDanial]

Two thoughts came to mind: try running cloc at the parent directory of your deep structure 1) from WSL or 2) using the "Git for Bash" terminal that comes with the Git for Windows installation. I wonder if these approaches ride on top of the Windows filesystem or have a different mechanism.