Jump to content
AL|EN

Cross-platform Infinity Engine Mod Package - brainstorming

Infinity Engine Mod Package file extension  

10 members have voted

You do not have permission to vote in this poll, or see the poll results. Please sign in or register to vote in this poll.

Recommended Posts

I've finally take some time to talk about this topic. When I take look at the amount of the BWS 'custom extraction rules' and 'custom code' which was written only because modder made simple packaging mistake and then went MIA, i decided that there won't be any kind of 'extraction' feature for PI unless there will be a well-defined standard for all IE mods. Having a well-defined, universal and cross-platform Infinity Engine Mod Package and the cross-platform tool for creation of such package would be wonderful. Definitely there are advantages for players and for mod managers. This general idea was originally posted by wisp at PPG boards.

Key points:

- the era of caring about package size on disk or download times are long gone
- there is no point for offering two separate zip packages for macOS and Linux if the only differences are weidu and few OS-specific tools resulting ~1MB of file difference
- macOS/Linux users offter are asked to download weidu, extract it, copy & rename and set executable bit. Example: http://www.shsforums.net/files/file/949-atweaks-platform-independent/ - just look at the 'Installation instructions' - it's horrible idea to require preforming all of those actions by macOS/Linux users. Those things must go away, thus the simple requirements for new package: it needs to have everything which is needed for mod installation.

Advantages:

  • standardization, no more rar, tar etc, all community created tools can support it without worrying about edge-cases
  • players will no longer need to install 3-rd party tools in order to extract mod, mod manager will do it
  • such mod packages can be double-clicked inside OS and associated program will be open and offer extraction (and possible installation)
  • it will allow to introduce 'infinity mod package protocol link' similar like Steam steam:// or Nexus Mod Manager nmm:// protocol links

The Infinity Engine Mod Package:// protocol link:

Those are special links on the web page, they look like this:

steam://run:<AppId> - if the user click at this link, his local Steam application will be executed and the game will launch or the user will be asked if the game should be downloaded.
nmm://ModID - if the user click at this link, his local NMM application will be executed and the mod will be automatically downloaded and added to mod list.

If you never have used them, please try via Steam or NMM/Vortex - it is a pleasure to see how everything works with one click. I want this to happen.

So Infinity Engine Mod Package:// protocol could be defined like this:

iemod://github.com/Gibberlings3/Tweaks-Anthology
  • player click at such link
  • associated application will open
  • mod will be downloaded and extracted (no overwrite)
  • additional option would be to offer installation after extraction but i'm not sure about this and how it will play

Support for such 'protocol' can't be established if there is no package definition because associated app must know ahead what's at the other side. If someone will point to the non-standard package either download/extraction will fail or extraction into game directory won't be correct so the installation cannot be started properly.

1. File format: ZIP

Quoting wisp:

Quote

it's a very widely supported format and more advanced compression algorithms and formats offer very limited advantages over ZIP since most mods consist of text, which compresses well regardless, and already compressed resources (images, audio), which further compress poorly, regardless. I tested this with a few mods and between ZIP, 7Z and TAR.XZ the biggest difference I saw was on the order of 100 kBi.

I agree that zip is very good choice, every OS has build-in support for it.

2. File extension: .iemod - Infinity Engine Mod Package

Different extension allows for custom file association for double-click action and extra actions for eg: Right-Click Menu: Install to> Game Folder. Besides, it's not only zip file, it's a zip contains IE mod with specific content/rules. Zipping mod files won't prevent package mistakes, modders need to use special mod packager utility.

The .iemod file extension is not the good choice, it's "Infinity Engine Mod Package", not "Infinity Engine Mod" source file. So how about .iemp as file extension?

If the 4-letter extension looks odd, how about .iep (but it's already used by other programs) or .iex (no application use such extension) ?

3. Filename:

Simple format: ModName-version.iemp, nothing fancy about it, mod name can be taken from metadata Name. Question: should spaces be allowed?
Filename doesn't mater at all under the condition that it doesn't contains forbidden special characters/folders:

Absolute minimum set of forbidden special characters:

<
>
:
"
/
\
|
?
*
'\0' (NUL)

Absolute minimum of forbidden special filenames:

CON, PRN, AUX, NUL, COM0, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT0, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, LPT9

cross-platform package simply cannot contain such special characters and cannot use such special filenames because those are reserved.

 4. The content of the package:

Across all those years, people put crazy things inside their mod packages: executable's inside zip, double compressed files like zip inside rar, override & backup folders, Windows 'thumbs.db' cache files and macOS '.DS_Store' folders etc.

So let's exclude some files and folders:

$RECYCLE.BIN/
__macosx
backup/
temp/
tmp/

.*
!/.gitattributes
!/.gitconfig
!/.github
!/.gitignore

*.bak
*.bat
*.iemod
*.lnk
*.rar
*.tar*
*.tmp
*.zip

Thumbs.db

all of those files/folders can be easy excluded at the creation time and should not be part of the final package.

5. WeiDU executable:

I would much prefer a solution where modder explicitly defines the weidu version which he used in order to create and install the mod.

But until we get to this point, there is a simple solution:

- weidu windows: .exe
- weidu macOS: no extension
- weidu Linux: .bin (please don't tell me that 'Linux don't have executable file extension', it doesn't matter)

this way modder can include all necessary files in order for mod to be installed. I will post universal '.command' file later.

6. Mod structure:

We need to eliminate possibility of double and tipple extra top-level directories, for mods: player should be able to extract such package into game directory and everything should work correctly. The way how to do it is to check the level of the tp2/tp3 mod definition file and fail if it's more that one-level.

7. Tools:

I've created packaging utility which is very similar to 'G3 ModPackage' tool but it's cross-platform and can produce .iemod packages and optionally pure .zip files, it's using ModID-version.iemod filename schema. It doesn't have validation yet but it's easy to implement. I will post it latter when I will do final touch.

That's all for now, please share you feedback.

Edited by AL|EN

Share this post


Link to post

EDIT - I really miss the button that lets you see the post formatting, so I can fix stuff like the badly-formed quote below.  Right now, in 'edit' mode, I can't get a cursor inside the box to cut the last sentence and paste it elsewhere.  In fact I can't even select text to copy it.  What on earth is wrong here??

I agree that having separate downloads for Win/macOS/Linux is unnecessary and kind of silly.  I mean, unless the macOS or Linux package includes some actual platform-specific utility or program, but that is very rare these days.  (I think Sirene or Isra might still need special audio handling, but even that could be fixed if the author wanted to.)

Moreover, platform-specific packaging can actually be harmful.  Sometimes I want the Windows version of a mod, but when I download an .exe file I can't get inside it to just look at the files in the mod folder.  That makes me sad.  :( And by "sad," I mean "angry."

I really think simpler is better.  There are 3 objects in my mods: 1) the mod folder; 2) a Windows Weidu executable; 3) the Mac Weidu Launcher, which includes the macOS Weidu executable.  If Windows users are lazy and extract the whole thing into their game folder, the Mac Weidu Launcher will overwrite itself and the total number of files will be equal to the number of files they get if they download Windows-specific versions of the mods... plus one.  For the benefit of including Mac compatibility with every mod, I think a single extra file in players' game folder is a very acceptable price to pay.  I am obviously biased, but for the utter simplicity of how it is packaged and how it is extracted and used, I think this is a good model.

If you want to add some chrome, for stuff like double-click extraction etc. that sounds dandy.  But you mostly spoke about the structure and bundling of mods - not their extraction.  

Quote

- players will no longer need to install 3-rd party tools in order to extract mod, mod manager will do it

- such mod packages can be double-clicked inside OS and associated program will be open and offer extraction (and possible installation)

What mod manager are you talking about?  What associated program will open and offer extraction?  On macOS and Linux, these things don't exist.

Edited by subtledoctor

Share this post


Link to post
Quote

- Linux users additionally need to download weidu, extract it, copy & rename and set executable bit - those things must go away

This is not true. Even if the user's distro does not provide a package for weidu, you still just have to download and extract it.

Quote

- players will no longer need to install 3-rd party tools in order to extract mod, mod manager will do it

The manager is a 3rd party tool. ;) Not much of an upgrade there, unless the mod was packed with something really exotic.

 

1. Not true, but not far from it. Can be ignored.

2. File extension is irrelevant, but why have it different, if it will just be a zip archive?

3. Why does the file name matter at all?

5. This sounds ridiculous — every mod would currently pack ~10M of extra crap. If people are supposed to use a mod manager, have that ensure it has all the necessary tooling. The packages can never be self-contained, since the payload is programmatic, so there's no point in exploding the sizes with redundant files.

@subtledoctorIt's up to the browser to implement, so at least for firefox the extension with a new protocol handler is possible — by the user.

Edited by lynx

Share this post


Link to post
3 hours ago, lynx said:

This is not true. Even if the user's distro does not provide a package for weidu, you still just have to download and extract it.

Maybe I wasn't clear enough: I refer to the problem of various macOS/linux mod packages like for eg: http://www.shsforums.net/files/file/949-atweaks-platform-independent/

Just look at the 'Installation instructions' - it's horrible idea to require preforming all of those actions by macOS/Linux users. Thus the simple requirements for new package: it needs to have everything which is needed for mod installation.

3 hours ago, lynx said:

The manager is a 3rd party tool. ;) Not much of an upgrade there, unless the mod was packed with something really exotic

You got me 😛 Yep, it's 3rd party tool but clean OS have 'Expand-Archive'/'unzip' commands which can extract .zip files atleas. Can't do this with rar/7z.

3 hours ago, lynx said:

2. File extension is irrelevant, but why have it different, if it will just be a zip archive?

3. Why does the file name matter at all?

5. This sounds ridiculous — every mod would currently pack ~10M of extra crap. If people are supposed to use a mod manager, have that ensure it has all the necessary tooling. The packages can never be self-contained, since the payload is programmatic, so there's no point in exploding the sizes with redundant files.

At 2. I thought that it was oblivious: different extension allows for custom file association and extra actions (for eg: Right-Click Menu: Install to> Game Folder). Besides, it's not only zip file, it's a zip contains IE mod with specific content/rules. Zipping mod files won't prevent package mistakes unlike using special mod packager utility.

At 3. Good point, filename doesn't mater at all under the condition that it doesn't contains forbidden special characters/folders. My example refer to how package tool would name it during creation, edited.

At 5. The 'exploding' is a bit exaggeration for 10 MB, don't you think? The size doesn't matter anymore, especially when the price is "go there, download this, extract, set executable bit and run". Sure, I can include weidu and other tools but what about people who don't want to use mod managers at all? Having separate package with weidu+tools and without them is pointless. I'm in favor of simple requirements for the package: it needs to have everything which is needed for mod installation.

Edited by AL|EN

Share this post


Link to post

Those mac/linux instructions only need to be done once; the same way you may need to install a non-free zip handler.

5. People who don't want to use your tool will use whatever they're using now. Only windows releases are currently bloated with weidu, so there'd be no change for the rest. If the modders want to support installs without the tool, they'll either have to complicate their instructions or still provide classical formats anyway.

And since it's not at all obvious, I'm not complaining because of slow internet, but because of environmental impacts. 😛

Share this post


Link to post

@subtledoctor 
If a mod is utilizing platform-specific tools (windows oggdec, bat files etc), it's almost likely that the mod can't be installed properly at macOS/Linux at all so indeed update is required.

The "Mac Weidu Launcher" thing is an addition to the mod just as "setup-modname.command" file is - a helpful thing. It makes no difference for Windows users but it is helpful for macOS users.

5 hours ago, subtledoctor said:
Quote

- players will no longer need to install 3-rd party tools in order to extract mod, mod manager will do it

- such mod packages can be double-clicked inside OS and associated program will be open and offer extraction (and possible installation)

What mod manager are you talking about?  What associated program will open and offer extraction?  On macOS and Linux, these things don't exist.

Those tool/features yet don't exist. Zeitgeist exist and such feature it's part of the roadmap. PI has cross-platform as long-term goal. But anyway, this fact it's irrelevant: such tool can't exist for Android and iOS but Windows, macOS and Linux OS are able to support double-click (+ extra actions) so we shouldn't limit solutions just because 'I can't utilize it on my OS yet'.

Edited by AL|EN

Share this post


Link to post

I'm going to treat this as a general design for the future, rather than something that needs to work now with current WeiDU or the mod manager you are developing.

We should move away from including platform-specific executables in mods, whether that's WeiDU, oggdec / sox, tisunpack, or tile2ee.  Users only need one copy of these tools, not hundreds. These tools should be handled by Zeitgeist or a mod manager.

ZIP is a fine choice for compression format, since many libraries support it.

A custom file extension allows easy integration with mod managers.  It makes the files slightly harder to open if you have no mod manager, but the future is in mod managers.

I think packages should be able to contain multiple mods. Each would have its own metadata file within the package.

Share this post


Link to post
On 6/19/2019 at 11:11 PM, Mike1072 said:

I'm going to treat this as a general design for the future, rather than something that needs to work now with current WeiDU or the mod manager you are developing.

What do you mean by that? The feature 'double-click > extract" feature is ready. Packaging utility is almost ready. PI downloads weidu already. Adding other tools could be next. All it's left is agreement on key points, publish instructions and that's it.

On 6/19/2019 at 11:11 PM, Mike1072 said:

We should move away from including platform-specific executables in mods, whether that's WeiDU, oggdec / sox, tisunpack, or tile2ee.  Users only need one copy of these tools, not hundreds. These tools should be handled by Zeitgeist or a mod manager.

I think that weidu itself must be 'versioned', other tools like ogg, iconv rather not.

Edited by AL|EN

Share this post


Link to post

2. Is this not just you basically saying it's so. I might as well say it's an IE mod, hence .iemod. It also has the advantage of being  less letter salad. You seem to suggest the distinction has something to do with "source" files, but there's meaningfully no such thing. The "source" files are the mod.

5. Regarding the proposed extension for WeiDU-Linux: absolutely not. WeiDU has been weidu since the beginning (or at least, since it was  lowercased by default) and changing it would break all existing scripts and tooling that relied on it. GNU/Linux conventions are overwhelmingly that executables have no extension. You don't get to impose a new convention because you think it would be more convenient.

6. Including copies of WeiDU in every mod was ridiculous in the 1990s, it remains ridiculous today and it would be pants-on-head silly if "mod managers" were  to get involved. The "mod manager" can simply include or download the appropriate version of WeiDU and essentially have it all handled behind the scene. Zeitgeist already does much of this.

 

As a side note: ALIEN, have you investigated the possibilities for cross-platform support for your tool? I'm not at all knowledgeable about C#, mostly because while the language is cross-platform, the libraries and frameworks you'd use to create a Windows program were not. The corresponding stuff that were available for GNU/Linux is obsolete on Windows and on macOS you'd use something different that only worked on macOS. But maybe that's changed by now.

 

Sweet dog is the quote button an exercise in frustration now!

Share this post


Link to post
2 hours ago, Wisp said:

2. Is this not just you basically saying it's so. I might as well say it's an IE mod, hence .iemod. It also has the advantage of being  less letter salad. You seem to suggest the distinction has something to do with "source" files, but there's meaningfully no such thing. The "source" files are the mod.

Yes, I suggest the distinction between 'infinity ending mod' as tp2/tp3 and a package with the mod itself. The 'less letter salad' is a valid argument. It dosen't really matter, all I need is agreement of how it will be called.

2 hours ago, Wisp said:

5. Regarding the proposed extension for WeiDU-Linux: absolutely not. WeiDU has been weidu since the beginning (or at least, since it was  lowercased by default) and changing it would break all existing scripts and tooling that relied on it. GNU/Linux conventions are overwhelmingly that executables have no extension. You don't get to impose a new convention because you think it would be more convenient.

It was suggestion how a modder can include 3 weidu executable (+ .command file) inside mod! Not how weidu linux should be called, especially when we are consider the idea of mod managers having those executable included. Then it's weidu.exe for Windows and weidu for macOS/Linux.

2 hours ago, Wisp said:

Including copies of WeiDU in every mod was ridiculous in the 1990s, it remains ridiculous today and it would be pants-on-head silly if "mod managers" were  to get involved. The "mod manager" can simply include or download the appropriate version of WeiDU and essentially have it all handled behind the scene. Zeitgeist already does much of this.

Project Infinity also doing this. It's fine for me. I can download/include any kind of tool. Weidu would be copied into game directory. Where the other tools should be placed? Also there or maybe something like <GameDir>\Tools\ ?

2 hours ago, Wisp said:

As a side note: ALIEN, have you investigated the possibilities for cross-platform support for your tool? I'm not at all knowledgeable about C#, mostly because while the language is cross-platform, the libraries and frameworks you'd use to create a Windows program were not. The corresponding stuff that were available for GNU/Linux is obsolete on Windows and on macOS you'd use something different that only worked on macOS. But maybe that's changed by now.

The c# has AvaloniaUI (c# MVVM ) and QML.Net (QT binding) which works on Win, macOS and Linux. There is also Python + QT as a valid solution for cross-platform since Python is very similar to Powershell. But what is the reason of bringing this matter now? What's the problem?

I really need you stance at the filename and package content validation.

Edited by AL|EN

Share this post


Link to post
2 hours ago, Wisp said:

As a side note: ALIEN, have you investigated the possibilities for cross-platform support for your tool? I'm not at all knowledgeable about C#, mostly because while the language is cross-platform, the libraries and frameworks you'd use to create a Windows program were not. The corresponding stuff that were available for GNU/Linux is obsolete on Windows and on macOS you'd use something different that only worked on macOS. But maybe that's changed by now.

I think Alien's tool is currently written in PowerShell, not C#.

To answer your question, .NET Core has made great strides in every area except a GUI framework.  The 3rd-party Avalonia UI that Alien linked seems to be the closest thing available, but it's in beta and could be a struggle to use.

You can write cross-platform console applications, web applications, and web APIs with .NET Core.

If I were to make a GUI application, I'd probably write the logic in .NET Core libraries and then create a Windows-only WPF project using the libraries.

Share this post


Link to post
5 hours ago, AL|EN said:

What do you mean by that? The feature 'double-click > extract" feature is ready. Packaging utility is almost ready. PI downloads weidu already. Adding other tools could be next. All it's left is agreement on key points, publish instructions and that's it.

I just meant that my response discussed a design that may require changes in mods, changes in WeiDU, changes in Zeitgeist, and changes in any mod manager that may want to interact with mods. It was not concerned with the practicality of a solution that could be implemented in your tool today.

I think that

weidu itself must be 'versioned', other tools like ogg, iconv rather not.

Is future WeiDU going to abandon backwards compatibility?

Share this post


Link to post
1 hour ago, Mike1072 said:

I think Alien's tool is currently written in PowerShell, not C#.

To answer your question, .NET Core has made great strides in every area except a GUI framework.  The 3rd-party Avalonia UI that Alien linked seems to be the closest thing available, but it's in beta and could be a struggle to use.

Yep, mostly Powershell which is also cross-platform and open source. The cross-platform GUI part will be done when all major features will be implemented and tested (we can see what's work and what's not), the one which left are EET suport and conflicts + dependencies.

46 minutes ago, Mike1072 said:

I just meant that my response discussed a design that may require changes in mods, changes in WeiDU, changes in Zeitgeist, and changes in any mod manager that may want to interact with mods. It was not concerned with the practicality of a solution that could be implemented in your tool today.

If this would require some major changes for mods, I would probably scrap the idea. But the changes are main at the packaging tool/distribution. 

46 minutes ago, Mike1072 said:

Is future WeiDU going to abandon backwards compatibility?

Not at all. It's a new feature for modders to pin weidu version so they will be sure that new weidu version won't break/change how it installs/act. Then, there is no more of 'backwards compatibility' because mods operate at the defined level of features/functions which won't change unless modder want's implement new feature. You can read more about it at the PPG Roadmap 2 'Stack management'.

Edited by AL|EN

Share this post


Link to post
On 6/21/2019 at 11:17 AM, AL|EN said:

Project Infinity also doing this. It's fine for me. I can download/include any kind of tool. Weidu would be copied into game directory. Where the other tools should be placed? Also there or maybe something like <GameDir>\Tools\ ?

Zeitgeist keeps WeiDU in the same directory as itself, which could be anywhere the user has appropriate permissions. On account of the version-pinning thing, I am considering more complicated systems, since we'd no longer be talking about 1 WeiDU executable; rather, it would be 1 for each version in play. I propose the location of WeiDU be left as an implementation detail of the front-end, but I would suggest it's in some location that is user-wide, rather than game-wide. If there is benefit in consolidating in the future, it would be safe to do so then.

 

On 6/21/2019 at 11:17 AM, AL|EN said:

But what is the reason of bringing this matter now? What's the problem?

There's no problem. You mentioned cross-platform upthread and I got curios.

On 6/21/2019 at 11:17 AM, AL|EN said:

I really need you stance at the filename and package content validation.

If I get to decide, I say .iemod.

I have no real objections to those restrictions on filenames or package contents, but I see no reason to prohibit override/, tmp/ and temp/. As long as it is not $GAME_DIR/override, it seems kind of arbitrary.
I suggest dot-files (and directories) should also be filtered out, or at least the ones used by git. Off-hand I can't think of a good reason for wanting to include dot-files, and if there are any present in the mod tree, chances are they are not intended for distribution in this context.

 

On 6/21/2019 at 2:07 PM, AL|EN said:

Not at all. It's a new feature for modders to pin weidu version so they will be sure that new weidu version won't break/change how it installs/act. Then, there is no more of 'backwards compatibility' because mods operate at the defined level of features/functions which won't change unless modder want's implement new feature. You can read more about it at the PPG Roadmap 2 'Stack management'.

That is not quite an accurate assessment. Backward compatibility will still very much be a thing. Version-pinning would be optional, so all unpinned mods would need WeiDU to remain backward-compatible. It would allow those modders so inclined to get off the ride, as it were, and remain on some specific version indefinitely. Very hypothetically, it would also allow breaking changes to be made, since you could branch off the last non-breaking version and put it on life-support; breaking changes would be opt in. However, it's both exhausting and infuriating to be working against something that makes frequent breaking changes, so I wouldn't expect a great many of those, in any event.

Share this post


Link to post
Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...