Use hardlinks for common files in modlist installs

[Hirato]

Background

When installing multiple big modlists, particularly for the likes of Skyrim SE and Fallout 4, we are often dealing with 100GB in downloads, and then some 180GB+ in install size.
The downloads can already go into a shared folder, but this is in essence a similar idea, but for the installs themselves.

These installs often include a full copy of the game, via the 'Stock game' feature, and also many gigabytes of common mods.
I think it would be good if the copy of the base game can use hardlinks, the net effect being that the first modlist I install has to pay the full disk cost, but the 2nd (or any further subsequent) modlist, is able to determine that it has 130GB of files in common, hard link them, and then install the 50GB that is unique.

What is a hard link?

Hardlinks are a file system feature supported on all major operating systems, including Windows since NTFS supplanted FAT32.
Basically, it allows you access the exact same data from a different path on the file system.
For example, you could have a 2.6GB archive, normally to have this accessible in 4 locations, you'd have to make 4 copies of it, costing you 10.4GB of disk space.
But with a hardlink, you can link the file to those other locations, so you have a single 2.6GB file, that's accessible in 4 spots.

How WJ could perhaps approach this

I have my Fallout 4 modlists located at: Z:\Mod Organizer\Fallout 4\Wabbajack
With my personal MO stuff, and the shared downloads directory in the folder above.
But let's just talk with respect to that folder I made for the FO4 modlists.

So let's say I install my first Modlist to .\ModList1
One option is for WJ to then make a hidden folder next to it, let's say .\WJ Shared Install
WJ will extract the files it intends to install there, using perhaps a similar structure to the temp folder; preferably this should also pick up unchanged files between small mod updates.
After extracting the files here, including making a full copy of the game for the Stock Game feature if applicable,
WJ should then install the modlist by hard linking the files from here to that target path.
This should only be done using a whitelist of file types/extensions, with ones outside the whitelist being installed the traditional way.
If I then install another modlist to '.\ModList2` it should pick up that same folder, and link data were possible.

For fallout 4, file types for consideration should include .exe .dll .ba2 .bk2 .swf .nif .dds .hkx .wav .lip .xwm2

These Shared Install Folder are probably best off being game specific, while not impossible (e.g. Skyrim and SkyrimSE), there is very little chance of there being matching files.

After a successful install, WJ should vacuum this Shared Install folder, and delete anything that has only a single link remaining; these files have basically been orphaned.

If this Shared Install folder is deleted manually, WJ should also be able to repopulate it by updating/reinstalling a modlist and successfully finding matching files already in place. It should also be able to consolidate cases where the Shared Install data and the install has the correct file present, but the two are copies and not a link.

Should this be done for everything?

The short answer is absolutely not.
This approach has one major flaw; the file systems commonly used have no Copy of Write (CoW) functionality. For windows this comes with ReFS, and unless you're using a funny enterprise/server edition of Windows, you're not able to actually make partitions with this FS. For Linux it's a bit easier, but the common ext4 does not support it either.
Files that should be excluded should be anything that's:
a) list specific tool output (e.g. xlodgen: .btr .bto)
b) extremely likely to be modified by a user to their liking (e.g. plugin config .ini .xml .json .toml .txt etc)
c) files only the modlist authors are likely to customise (For FO4 this would include plugins and masters; e.g. to remove bad edits)
d) tool automation (e.g. xedit scripts).

If there was a CoW flag, making an edit to any of these files will implicitly make a copy and unlink it from the others; in other words you're free to edit it as you please.
However, without this flag, if you made a change to one of these hardlinked files, it will be reflected everywhere else too. This means you may unwittingly modify all your modlists, and not just the one you thought you were interacting with.

Exceptions should be supported

There should certainly be support for exemptions.
For the Stock Game feature, there is certainly merit in hardlinking the ESM files here too.
But there's also merit in sharing some configuration and other data.
For example with Skyrim SE, hardlinking the ini for SSE Display Tweaks would actually be a feature.

[github-actions]

Marked as stale due to no interactions in the last 90 days. If the issue still exists, please provide any possibly still needed information or excuse the devs for not getting around to addressing this issue yet. If this issue is still considered unsolved a dev will soon remove the stale tag from this issue.

[github-actions]

Closed due to inactivity. Please create a new issue if this problem has come up again and no dev has reopened this issue after a week of it being marked as closed.

[EzioTheDeadPoet]

re-opened cause it would be a cool storage saver, but it also got lost with time because it is a very heavy rewrite no one wanted to attempt.

[UhuruNUru]

@ EzioTheDeadPoet
You can easily do this yourself using LSE (Link Shell Extension) makes creating multiple links trivial.
Visual Indicator (File/folder overlays) of links, Properties Tab for Link details.
Best of all Drag ‘n’ Drop/Context Menu Link Creation, including Full cloning of Folder contents.

[EzioTheDeadPoet]

Setting it up is not the problem.
The problem comes from preventing user error like deleting a Modlist that was the source for another list. Or if you want a separate common files location that would require tracking and scanning all Modlists every time and keeping track of them to match common files to the common folder. And it would force us to make uninstalling lists more complicated for us and for users as well since if a file in the common files would not be used by the other files anymore we would need to detect and delete that.

So while the technology exists and is amazing it is too complex to maintain for this use case.

[Hirato]

I assume you're reflecting on the LSE tool.

I did consider all those points when I formulated my initial request; from explicitly using hard links (and not symlinks), to using a shared install folder with hashes for names (this avoids needing awareness of other modlist installs and allows quick lookup) adjacent to the modlist, etc.
It was all to keep the feature manageable, to minimise surprise to users, and avoid horrifically expensive and undesirable pitfalls.

LSE looks neat, but it doesn't seem to have any automatic de-duplication functionality - let alone selective de-duplication.
Whatever shared common data we can hard link with LSE, these links will only happen through a long, involved, tedious, and very manual process.

[github-actions]

Marked as stale due to no interactions in the last 90 days. If the issue still exists, please provide any possibly still needed information or excuse the devs for not getting around to addressing this issue yet. If this issue is still considered unsolved a dev will soon remove the stale tag from this issue.

[Hirato]

Not to sound conceited, but I'd appreciate someone higher up the food chain at least saying something like "We're not interested in such a feature at this time", rather than just letting it time out. :)

[EzioTheDeadPoet]

I still like the idea, but time and motivation to work on big rewrites on the app have run out for many of us for various different reasons.

Personally I always preferred working on solutions for issues I found to come up in my workflow of using the app. So already this feature is low on my personal priority list of things. I renewed it anyway in case a new first time contributor feels like they want to tackle the challenge or one of the older contributors might see it and wants to try it. But since no one for a long time decided to be interested in this I decided to not do it this time.

Also recently I was the victim of a targeted attack because I am part of this project that also made me lose interest in investing any time into it for the sake of others unless I see fun in the challenge of the project. And sadly for this request I don't see fun in rewriting the whole app for this feature.

This is a bit of a long answer but I hope it gives you the context and update on the situation you were looking for.

For now I will leave it open, but I probably won't renew the expiration timer again.