Possibility of hosting a fork of 7even's endless-ships with an auto-build that generates the Encylopedia in the Endless Sky website?

[Zitchas]

Currently, there is a website called the "Endless Sky Encyclopedia." It's an extremely useful reference for Endless Sky's ships (including built-in ways to filter based on factions and categories), along with outfit lists that can be sorted conveniently.

Currently, two versions exist:
http://endless-sky.7vn.io/
http://dev.endless-sky.7vn.io/

The first one was historically kept up to date with Endless Sky's stable release.
The second one was updated periodically to continuous.

Unfortunately, 7even is busy and/or otherwise hasn't been able to update it in a while. So I have two questions:

1. Given all our build scrips can generate the Endless Sky on GitHub, would it be possible to setup something similar within a fork of "endless-ships" that runs through it's build process on GitHub?
2. Would people be interested in having it hosted on the Endless Sky website? Perhaps even setup to be automatically triggered when we do releases and/or merges (for the stable and dev versions, respectively)

It is an extremely useful resource, and much more user-friendly than pointing people at spreadsheets, and its appearance is such that it would probably fit right into our website with minimal effort. Ideally, I'd like to see it on https://endless-sky.github.io/ , probably listed between the link to the "Manual" and the link to the "Wiki" on the left sidebar.

(As for rights/usability - the fact that it has a "fork me" with a link to the github repository and instructions on how to build it are probably fairly clear indicators that people are welcome to do so...)

The pages in question both link to https://github.com/7even/endless-ships

Which includes full instructions on how to build it. From the instructions, it appears to build a complete website that just needs to be uploaded somewhere. Unfortunately, I'm not familiar with boot-clj or yarn, and lack a webhost to try it on.

@quyykk You've been mentioned as one of the people who'd probably best know if this is feasible.

@7even As this is your creation, we'd love to have you stay involved with this and offer your opinions/thoughts on it. Your tool is too good to keep hidden or let it fall into irrelevancy due to lack of updates. Thank you for making it!

[warp-core]

@Hecter94 may also be interested/have something to offer.

[quyykk]

@quyykk You've been mentioned as one of the people who'd probably best know if this is feasible.

This is certainly feasible, since endless-ships seem to parse the data files to create the tables. We can easily create a build job on the website's repo that runs daily, downloads the repo, updates the tables and publishes.

Would people be interested in having it hosted on the Endless Sky website?

I think that's a good idea! (On an unrelated note it seems like the boot-clj is not maintained anymore, but oh well). The code is GPL so it's not an issue.

[Zitchas]

Great! Is there anything I can do to help move this along? Obviously I don't know anything about creating the build jobs, but if I can help with anything else there, I'd be happy to.

[Hecter94]

I prefer spreadsheets, simply viewing the data is generally not enough to do any work with; most of what I use tools such as these for, and what I've seen most others use them for, is directly comparing outfits/ships and using them as tools to balance new content against existing content.

To that end, you need the ability to edit the data and build your own functions to insert in. As far as I'm aware, the tool created by 7even cannot do that.

There's a lot more information available in the spreadsheet as well, which I think is information that a lot of your general players will find useful, like Combined HP, Fighter/Drone Bays, Max Bunks, Loaded Cost, etc.

One question I have for @Zitchas, you mention

and much more user-friendly than pointing people at spreadsheets

Could I ask what's user-unfriendly about the spreadsheets we have? They're hosted online like 7even's tool, and the layout and functionality seem very similar between the two.

7even's tool seems to be mimicking the look, style, and functionality of a spreadsheet:

And they're both viewed/sorted the same way.

One difference I note is that the way 7even has the filtering set up seems more user-friendly than what I have.

Though mine can also filter by any value in any column, it's a more powerful tool at the expense of that user-friendliness.

With all that said, I feel like the spreadsheet is just better; it has more data, more options, more customizability, seems similarly user-friendly (in my opinion), and even looks very similar to 7even's tool.
I don't feel we'll be able to replace the spreadsheets since they are extremely valuable for content creation, much more so than the encyclopedia. Still, they come with a bit of a learning curve and aren't quite as user-friendly despite being more powerful.

It seems like there are two options, then:

1. Keep the spreadsheets as development tools used by the people who appreciate the additional power and flexibility offered by spreadsheets and don't mind the extra learning curve. In contrast, the Endless Sky encyclopedia could be a much simpler tool for your general user who wants to view the data and not do anything more advanced.
2. Pick one of the two options to use as the default, and implement functionality within it to allow it to fill both roles. This means either updating the spreadsheet to be more user-friendly or updating 7even's tool with the additional tools that the spreadsheet comes with.

Of those two options, I'd lean towards the second option as I think having a single tool with all the functionality we need is better than maintaining two separate ones that cover two different use cases. Though that depends on whether it's possible to update the spreadsheets to be user-friendly enough or update 7even's tool with all the functionality spreadsheets provide.

[Zitchas]

Question: How much do we want to adopt EVE-online's reputation as "Spreadsheets in space?"

Your spreadsheets definitely are more useful for devs and people that need to crunch data (read: balancers and min-maxing optimizers). But I find 7even's to be vastly friendlier to look at. Yes, maybe not as easy to do ship to ship comparisons, but for people who are more interested in browsing ships (and, worth noting, all the variant configurations can be acessed from the ship's individual page), it's a much more friendly and approachable experience.

To put it another way: I don't see 7even's website and your spreadsheets as being rivals. They each have their own target audience, their own formats, and their own strengths. I don't see any reason why we can't have both so that their respective audiences can benefit from them.

To be honest, if a regular player asked me for somewhere I could point them for information on ships in the game, they're getting pointed to 7even's website. It's not even a matter of debate. They have a basic table that compares the main fields, and more importantly they can click through to actually see the ships, along with the ship descriptions and variants that they might run into. As far as gameplay experience goes, this is so far beyond a spreadsheet that I wouldn't even bother comparing the two. I probably wouldn't even mention that spreadsheets exist to the average player.

Now, if someone comes to me wanting something that will let them compare the absolute max values of ships to help them build the "ultimate" best performing ship and they need to be able to compare outfit to mass ratios or do other deep dives into the data. Yeah, I'll point those people to a spreadsheet without hesitation. I've never even seen yours up to now, but sure, I'll give it a look, and maybe I'd direct people to it, or maybe to Amazinite's.

So, for the general user base, 7even's provides all the functionality they need, and does so in a very appealing, very accessible, easy to use format, and if we had to pick just one thing, I would pick it because what we put on the website alongside the player manual needs to be something that is tailor-made to be useful for and make life easy for the general mass of players.

---

Now, we have two other aspects to consider:
a) ease of maintnance.
b) versioning.

Ease of Maintenance

7even's website literally has a generator. As best I understand it, it simply needs to be pointed at a copy of our repository and run, and it gives us a complete website. From my conversations with 7even, the only time it needs anything more in-depth than that is when we add new factions or categories to it. Which, honestly, doesn't happen very often. All the rest of the churn is handled automatically.

Spreadsheets... I don't know. Do you have a fully automatic script that handles updating it? Or is it a mix of manual entry and copy/paste?

Versioning

7even's website actually has two versions. The regular one, and the one with the dev. prefix. The former was synced to stable, and the later periodically updated to continuous and/or unstable. Following this pattern we could easily do the same: Have 7even's follow the stable release (most useful for people stuck on regular releases) while we have a Dev one elsewhere that is continuously updated to follow continuous, and is a spreadsheet or otherwise has the greater data depth that Devs and the like need.

Although having said that, splitting it up that way kind of skips using the ease-of-updating we have in 7even's website generator for the area that needs the most updating.

Combining the two

For presentation and aesthetics, I think it would be best to stick with 7even's. All else being equal, it looks better than a spreadsheet. While it could be useful to have the additional functionality of the spreedsheets added to it, I'd also be cautious about overload. Most players don't need or want the same level of details that a Dev does. That being said, I'm sure there is a fair bit of functionality that could be added to 7even's website that would be useful/beneficial for everyone. (for instance, having faction filters on the outfit page)

---

So, all considered, I'd probably just opt to have two versions of the website for inclusion in the website, and the spreadsheet remains a spreadsheet for the use of the Devs that need that additional detail.

[Zitchas]

addendum: I actually don't know where your spreadsheets are. I haven't seen a link to them, either. The pictures... look like spreadsheets.

[Hecter94]

(and, worth noting, all the variant configurations can be accessed from the ship's individual page)

Okay, this is cool. I didn't know you could click on the ship names to see more detail about them. That's nice functionality that the spreadsheet does not have which I wasn't aware of, which I do agree offers something very appealing for your general user, and is probably something that I'd have no way of replicating in the spreadsheet.

To put it another way: I don't see 7even's website and your spreadsheets as being rivals. They each have their own target audience, their own formats, and their own strengths. I don't see any reason why we can't have both so that their respective audiences can benefit from them.

I suppose it comes down to: "Do we want to maintain two separate versions of the same tool for two different audiences? Or work on a single tool that can fulfill both purposes for both audiences?".
I'd prefer we have a single tool for ease of maintenance and in the interest of not splitting our development efforts.
Whether that's updating 7even's with the additional functionality that devs need, or updating the spreadsheet with the general accessibility and ease of use that general players need.

So, for the general user base, 7even's provides all the functionality they need, and does so in a very appealing, very accessible, easy to use format

What could be done with the spreadsheet to make it more appealing, accessible and easy to use? They're both hosted online, they both look very similar, and they both function very similarly.
Even outside this current discussion, I'd like to know so I can see about updating mine to be easier to use just in general, if you have any specific things you think should be changed about how it's presented or used.

Now, we have two other aspects to consider: a) ease of maintnance. b) versioning.

Ease of Maintenance

Spreadsheet uses the built-in game data printing to export all of the relevant info from the game, which is then uploaded to gsheets, all formatting and formulas are handled automatically in gsheets once the raw data is uploaded.
Currently I do manually paste the raw data in, but this is something that could be automated with a script to take the data and upload it to gsheets automatically.

The only parts that have to be manual are adding in Faction, Manufacturer, Method of Obtaining and Tier. Information not available in game data.

Versioning

This would be quite easy to do with the spreadsheets, could have one version which is on the last stable release and one which is on continuous.
Hell, we could have one for every major release going forward and just keep all of them available if someone ten years from now wants to go back and look at this version. Wouldn't be difficult to do since they aren't going to need any maintenance or updating.

Combining the two

For presentation and aesthetics, I think it would be best to stick with 7even's. All else being equal, it looks better than a spreadsheet. While it could be useful to have the additional functionality of the spreedsheets added to it, I'd also be cautious about overload.

Would it be possible to use 7even's and have a toggleable "expert mode", so to speak? With all of the extra functionality that devs and advanced users need, but hidden by default so it's not overload.

[Hecter94]

addendum: I actually don't know where your spreadsheets are. I haven't seen a link to them, either. The pictures... look like spreadsheets.

I've sent you the link to them previously
https://drive.google.com/drive/folders/1IyhujotU21aab_SlVlh7EzSbWDuDlnaH

[7even]

I think that running the site generator from a github workflow is a great idea.

There is a recent PR that adds Docker support which may simplify the process of building it if you're okay to deploy it as a Docker container; though it is targeted at people who just want to build the site for the most recent release of the game.

The manual building process is also pretty straightforward but right now it requires installing a lot of different tooling which is not exactly necessary. I want to get rid of both boot-clj and yarn since boot-clj is kinda abandoned and can be replaced by clojure's own tools.deps for the generator dependencies and tasks; while frontend part of the site is being built with shadow-cljs that can be used along with regular npm - no yarn needed here.

The ability to compare ships (or even outfits of the same category) side-by-side was always near the top of my wishlist for endless-ships but it requires some time to think through and implement. Another feature I would love to see there is some kind of a ship constructor where you can select a particular ship, then fill it with outfits of your choosing and see the resulting stats. These two features could even be combined into one screen so that you can compare 2 different ship builds.

Now to the more down-to-earth problems.

Ease of maintenance: yes, when a new faction is added to the game, it needs to be added to the site generator code (otherwise its label will look like white text on a white background, i.e. invisible). Everything else depends on the data extracted from the game's data files, so no other manual work should be needed.

Versioning: it would be great to have the ability to switch between any game versions but I'm not sure that modern versions of endless-ships will work with old versions of the game. Once again, factions are the source of the problem - there is no link between a ship/outfit and a faction that produces it in the game data. The only way to get it is to make some mapping between names of game data files and corresponding factions - this is how ships' factions are determined right now. But file names can change between the game's versions, and I wouldn't really like to track filenames over all versions of ES.

One solution to this problem would be to introduce some kind of a convention, like "all ships and outfits described by files inside the data/korath directory are supposed to belong to the Korath faction" (by the way it seems this is already the case - please correct me if I'm wrong); then the site generator could simply determine the faction from the filename. But once again, this will only work with game versions where these conventions were applied - there won't be a way to time-travel to 0.9.8 and see a correct list of ships with their respective factions.

With all that being said, I think I will continue working on the project, so I suggest keeping from forking it for now.

[Zitchas]

Sorry for taking a bit to reply.

Thank you for the information about the build process. It's a bit above me, but I hope that the additional detail is useful to the members of our team that handle the build processes.

In terms of the game files, in the short term using the convention that the folder they are in would be a good way to categorize which faction they belong to. There is one or two places where that could generate inconsistencies, with the void sprites currently listed in the Drak folder; but that very placement is something of a foreshadowing or hint of the way things are in the game that there is a connection between the two, so I don't see that as actually being a problem. Just an oddity.

In the long term there has been discussion about adding attributes to ships and outfits that specify their faction and/or shipyard of manufacture. In this case, shipyard being the brand of the ship, not the planet. That would probably be much more specific and safe to use, and less prone to errors. But we aren't there yet.

I don't know if desktop-side examples or inspiration is helpful for you, but if it is, there's this project:
https://github.com/robot-rover/endless-sky-viewer
It has a lot of similar ship browsing capabilities to the Endless Sky Encyclopedia, but it also includes a ship building element too.

[xX-Dillinger-Xx]

What has happened on this front? I use 7even's tool (dev version) all the time. In my opinion, I would use his tool over a spread sheet 90% of the time. However, spread sheets can be made to be much more user friendly, but most creators like the spread sheet look, and are very comfortable with it, so they don't take the time. I agree that there may be a few bits of useful data missing from 7even's, but it is real close to be 100% and would not require to much work to add the extra data columns. As an added benefit, I have discovered some new game content I was unaware of just by looking at the tool. I saw some new weapons that suddenly appeared after his latest update and then went to investigate in the game. I mostly use the sort feature to find which outfit that will "just fit" into the space I have. I really hope someone pursues this further.

[Zitchas]

It's been a while so my memory may be faulty, but if I recall correctly, it is basically as 7even said in their post:

With all that being said, I think I will continue working on the project, so I suggest keeping from forking it for now.

So long as they are working on it (and they still are) and maintaining it, I think it works quite well where it is. If they should ever cease to do so and/or hosting becomes an issue, then we can reconsider.

I suppose a shorter way of saying it would be "We'll follow 7even's lead on this."

[ei-grad]

Just to mention here, endless-sky command has -s flag, which allows to get data in CSV format for the spreadsheet (or a similar web page generation). Just run endless-sky -s --loaded --variants > endless-sky-ships.csv and open endless-sky-ships.csv in Excel, Google Sheets or Libreoffice Calc.

For other flags see a full man page https://manpages.ubuntu.com/manpages/noble/en/man6/endless-sky.6.html (source is a bit unreadable on Github).

I guess participants of this discussion know about this feature, but I came here from the search engine results page, and it will be nice to have a note about where related data could be acquired, such a great idea to include its export to the game binary.