Path handling is case sensitive on Linux

[qsniyg]

There are a lot of errors due to the path casing being incorrect.

For example, it was unable to find Skyrim.esm, as it was named skyrim.esm (due to an update script as a workaround for ModOrganizer 2 not working under wine).

This also causes problems with the actual ordering not working at all, other than for esm/esps that are completely lowercase.

Would it be possible to create a function that would look through files in a directory for a file with the same name, but lowercase both, for non-windows users?

Thanks by the way, for supporting linux, it's seriously appreciated!

[Ortham]

Case insensitivity is very complicated, and it general the only correct thing to do is to leave comparing paths to the filesystem, because NTFS's case insensitivity rules don't necessarily match whatever case insensitivity rules your code runs with, and the rules may have additional complications like being locale-dependent.

I probably need to review LOOT's current handling for consistency, but in general I think it makes sense that filesystem interactions (e.g. "does this file exist") should treat paths as binary blobs, and leave it to the filesystem to handle any case insensitivity, and any other use of plugin filenames, though they're part of paths, should treat them as text and handle them case insensitively.

This approach probably isn't applied consistently, and fixing that may fix your issue or it may make it worse, but I can't say for sure. If you're running Skyrim in an NTFS filesystem and have the ntfs-3g driver configured to enable case-insensitivity, I'd expect things to run as they do on Windows.

[qsniyg]

NTFS's case insensitivity rules don't necessarily match whatever case insensitivity rules your code runs with, and the rules may have additional complications like being locale-dependent

I looked through WINE's implementation, implementing case sensitivity properly wouldn't be too hard: https://github.com/wine-mirror/wine/blob/ba9f3dc198dfc81bb40159077b73b797006bb73c/dlls/ntdll/directory.c#L2034

!memicmpW( buffer, name, length )

Where memicmpW works by:

if ((ret = tolowerW(*str1) - tolowerW(*str2))) break;'

So far so good, although tolowerW is where it gets a little more complicated.

I'll try to write a separate library for this, based on WINE's code. Would also be useful for other tools for linux that change things on windows.

Although writing the library will likely not be too difficult, I'm guessing the difficulty will be in integrating it within both LOOT and libLOOT, replacing all filesystem functions. I could try to write a patch for this as well though if you would like.

leave it to the filesystem to handle any case insensitivity

While I understand your reasoning, the problem is that the vast majority of linux users do not use NTFS, and use filesystems using case-sensitive filenames instead. This also causes "hidden" issues, where it appears to work fine, but it actually ends up silently disabling ESPs with improper casing from the load order.

Edit: After doing a little more research, apparently NTFS is case-sensitive, but Windows treats it case-insensitive, so using NTFS under linux would likely not solve the problem either.

fixing that may fix your issue

Testing from manually changing each filename to be the correct case, it did indeed fix the issue.

[Ortham]

Although writing the library will likely not be too difficult, I'm guessing the difficulty will be in integrating it within both LOOT and libLOOT, replacing all filesystem functions. I could try to write a patch for this as well though if you would like.

No, I can't see myself accepting something like that. It would be moving away from using the standard filesystem APIs to a more complicated system, purely for the benefit of a tiny subset of users running on Linux.

the vast majority of linux users do not use NTFS

The vast majority of Linux users also aren't trying to run a tool that is primarily developed for Windows users to help manage the mods they've installed for a Windows game on a filesystem that wasn't a target for any of the developers involved. You have chosen to go off the beaten path, to not use an environment these things were designed for. You're going to encounter behavioural differences.

I expect that LOOT against a Skyrim that's been installed in an NTFS filesystem that's mounted with the ignore_case option will not have this issue, and until I know otherwise, I can't justify making Linux-specific code changes to mitigate the issue.

[qsniyg]

It would be moving away from using the standard filesystem APIs to a more complicated system [...] I can't justify making Linux-specific code changes to mitigate the issue.

Fair enough. I tried to write a wrapper around the filesystem API, with little success. Although it would be possible, I wasn't able to figure out a way to do it that wouldn't require rewriting the entire API, something I'd rather avoid doing.

However, I did instead manage to create a library you can inject, and with that, you can use LOOT (and presumably most other applications as well) without problems.

The library is here: https://github.com/qsniyg/libwinpath. To use:

winpath ./LOOT --no-sandbox

[Infernio]

I'm currently using ciopfs for storing data that needs to be case-insensitive on a case-sensitive FS. It's an overlay FS based on FUSE, which I just slap on some folder, add an appropriate entry to my /etc/fstab, and then it just works™.

If you don't want to create a full NTFS partition, that project is a pretty lightweight replacement.

[Ortham]

They both look like good options to have, I've tweaked the title of this issue to help anyone else hitting it find your links, and I'll keep this issue open (I also still mean to review the consistency of path handling).

[ThisNekoGuy]

I don't necessarily think it's fair to say that "most Linux users aren't trying to run x tool for Windows so f* 'em," even if Linux users are smaller in population when the same can be said of Windows users who don't even mod games to begin with in this context...
Especially when Linux users are only particularly vocal about what they're doing when things go wrong because they want solutions. Not trying to downplay the effort required, of course, just saying that's a rather toxic viewpoint.

That aside, Loot does run in Wine, and with basically every Elder Scrolls and Fallout game already running "in an environment they were never meant to" with Wine almost perfectly if not perfectly well: it would be great if Loot had better support in this area

[Ortham]

I don't necessarily think it's fair to say that "most Linux users aren't trying to run x tool for Windows so f* 'em,"

Nobody has said that.

[ThisNekoGuy]

The vast majority of Linux users also aren't trying to run a tool that is primarily developed for Windows users to help manage the mods they've installed for a Windows game on a filesystem that wasn't a target for any of the developers involved. You have chosen to go off the beaten path, to not use an environment these things were designed for. You're going to encounter behavioural differences.

I just perceived it that way

[ThisNekoGuy]

Brain damaged reasoning

Honestly, even though that's a mean way of putting it, you're not wrong - that kind of thinking is contradictory :/

Intentionally supporting a Linux build but not wanting to add Linux code is really bizarre; even saying that one doesn't know how to assist with a Linux specific issue would have made more sense than a statement like that

[Ortham]

What a pair of messages to wake up to... It's one (admittedly bizarre to me) thing to disagree with how I choose to spend my free time, but it's another to communicate that disagreement so rudely. cppcooper has been blocked because I won't tolerate such completely unwarranted behaviour if I can at all help it.

I'm going to lock this, because there's been no constructive conversation since Infernio's comment over two years ago, but I'll keep the issue open because it is an issue, and there is information in the comment thread that may be useful to people who encounter it.