projectinfinity IEMod - standardized, universal and cross-platform Infinity Engine Mod Package

[lynx]

I'm not sure what the "bad thing" refers to. Either way, running weidu doesn't need a console, you just have to manually tell it when clicking (usually only once btw, double clicking is not the default to trigger actions) ... unless the mime type has been registered like Wisp already mentioned. In which case it works automatically and there's no need for per-mod hacks whatsoever. But yes, something has to do it initially and it's not clear what should bear that responsibility.

[subtledoctor]

On 7/16/2019 at 4:02 AM, AL|EN said:

Let me understand what you actually trying to achieve:

I'm not trying to achieve anything in particular.  Just to keep things as simple and easy as possible for the user.  The current versions of mods for MacOS are kind of a mess. 

In any event I've updated the MWL (MacOS Weidu Launcher, yes I'm going to starts using a super-cool acronym ) and I think it beats any other existing way to install mods.  So what interests me most is that the download file format includes an easy way for Mac users to put the mod folder, and only the mod folder, into their game directory.

 

EDIT - actually I can be clearer than that. I'm from a time when the underlying architectures were vastly different and lots of mods didn't work at all on Macs without lots of work adapting them. 

If it seems like I don't really care about the .command files you want to include with mods, it's because those are only ever going to be included with a fraction of the mods out there, and they aren't going to go very far to make things better for players on that platform. I think it's much more important to put efforts into making sure the underlying mods themselves actually work. 

Edited July 24, 2019 by subtledoctor

[subtledoctor]

Also, the way G3 mods preface their download filenames with "osx-" drives me fracking bonkers.  First of all, that's not even the name of the operating system anymore.  Second, and much more importantly, it disrupts the alphabetical file sorting in my downloads folder and make it hard to browse for stuff I'm looking for.  ("Where's the Glam Vrock NPC pack again?  Oh right, under "o" )

[AL|EN]

3 hours ago, subtledoctor said:

Also, the way G3 mods preface their download filenames with "osx-" drives me fracking bonkers.  First of all, that's not even the name of the operating system anymore.  Second, and much more importantly, it disrupts the alphabetical file sorting in my downloads folder and make it hard to browse for stuff I'm looking for.  ("Where's the Glam Vrock NPC pack again?  Oh right, under "o" )

Yep, it makes me angry too! Ii's the same story with 'setup-modname.exe' in order to modify selected mod components. Lucky, when you have one zip file for all platforms, it doesn't  need os prefix.

[skellytz]

Looking at the contents of the example cross-platform package, the Mac version of WeiDU is used for the "setup-modexample" executable; however, the readmes and the script ("runs-on: ubuntu-latest") indicate that the cross-platform package works on Linux. Will the Mac WeiDU magically run on Linux or is the player currently still required to manually download and replace the installer?

Edited December 28, 2019 by skellytz

[AL|EN]

@skellytz The "cross-platform" refers to .iemod package which doesn't have weidu executable. Mod manager will handle the installation by calling it's own provisioned weidu. When it comes to the .zip packages (or downloaded master branch as zip) and linux, nothing has been changed and I understand that this is extremely uncomfortable. If you have any idea how to make things better for linux users, feel free to share.

[Magus]

Sorry for chiming in so late, I don't know if it's still relevant, but anyway:

1. I think that .iemod thing is good in general.
2. I don't care about the extension specifically, it could've been zip as far as I'm concerned. I guess it's more convenient for Windows users to have a specialized extension, though.

3. I agree that there's no reason to pack copies of weidu and other tools within .iemod. It's specifically desigined to be consumed by mod managers, and mod managers can take care of the tools themselves.

4. I do not understand the fuss about including or not including weidu executable inside, as well as other files (tisunpack.exe, thumbs.db and whatever). I would rather think of this in terms of IFTT (if this than that). What are you going to do if "setup-mymod.exe is included", and why?
   - Not including it is not a problem, as far as I can see.
   - Including it is not also not a problem, as far as I can see. A mod manager doesn't have to use files that exist on filesystem.
   OR do you actually plan to abort mod installation because there are files you don't like inside? (Aren't you causing the problem you wanted to avoid in the first place??)
   @AL|EN has pointed out to me that some antiviruses throw a false positive on weidu. I say: this is a problem with these antiviruses. Not with weidu. If a user purchased an antivirus, and it throws a false positive, it's the antivirus' fault. It must be communicated clearly to the user, the company producing the antivirus, and all other involved parties. I shall not budge a single inch on this.

5. All the talking and agreeing is nice, but I do not see the actual file format definition. Not as an editable forum post, but as something versionable, like an .md doc on github, at least? The reason I'm asking is that I might want/need to add automatic packaging to Golem.

[Wisp]

On 1/16/2020 at 7:25 PM, qwerty1234567 said:

1. All the talking and agreeing is nice, but I do not see the actual file format definition. Not as an editable forum post, but as something versionable, like an .md doc on github, at least? The reason I'm asking is that I might want/need to add automatic packaging to Golem.

First draft: https://github.com/WeiDUorg/zeitgeist/blob/master/iemod_spec

Some issues remain:

1. A tighter specification of the ZIP format and allowed/disallowed features. I need to do some boning up.

2. I can see legitimate reasons for why a mod might want to include archive files in the mod package, and I really don't think we are at a point where we can safely exclude .bat files from mod packages, so table may-exclude may be trimmed a bit.

3. Possibly some mention of files/directories that differ only in case, or maybe ZIP already has that covered.

Edited January 19, 2020 by Wisp

[AL|EN]

@Wisp

1. Primary goal is compatibility with all/most OSs and unzipping tools so my vote goes for: Store (no compression), Deflate

2. Regarding .bat - are you thinking about legacy mods which uses .bat in order to handle ogg>wav audio? Regarding archives inside archives - it is probably the edge case and should not affect anything.

3. Zip can store case-sensitive files or even the duplication of the same files but during extraction, files are overwritten.

P.S You need to add .md file extension in order to get properly formatted markdown.

Edited January 20, 2020 by AL|EN

[Wisp]

2. That and TIZ stuff. IEMOD is platform-agnostic, but it seems unnecessary to demand the contents are, as well. Especially since it means legacy mods could not be repackaged without being re-coded. Archives - right, it is of no particular concern if the IEMOD contains any, and it's unlikely enough that it would.

4. Anyway, I've updated the spec with some technical details, notably I think we need to specify that the contents should be in UTF-8, or extra-ASCII characters won't translate across locales. Seems less restrictive than requiring everything to be in ASCII. Diff.

[AL|EN]

Updates that I gather over the last months:

• use MARKDOWN file format with lists, tables for better reading
• added 'case-insensitive' for all rules
• update list of forbidden top-level files and folders
• update list of forbidden special files and folders
• added  list of forbidden files and folders names starting with prefixes
• added examples
• replace 'may-exclude' with 'should-exclude' to encourage best practices
• added ".*" (all files starting with dot) to 'should-exclude'
• removed 'bgforge.ini' as it no longer exist, it was renamed to ".bgforge.yml" and it's covered by ".*" rule
• removed archives since modders put the source code of unmodified extra tools into mod packages
• removed 'ehthumbs.db' because redundant in 2021 - those files are created only when someone who develops mod has Vista (<0.8% marketshare) and open the mod directory via Media Center application

Remarks:

1. It's painful to see that package format would allow pollution of main game directory with files, let this thing go away.
2. Replace 'must contain tp2' with 'if exist' so IEMOD package could be used for 'non-weidu' mods.
3. I'm not entirely sure if we should disallow 'override' directory. Simple mods which contain only files could be zipped with top-level  'override' directory which contain fixed files and the install proces would be straightforward - extracting package content into main game directory and that's it.

 

Quote

Version 0.11.0

The IEMOD format is intended to be a platform-independent distribution format for modifications for games using the Infinity Engine.

An IEMOD file:

• must be ZIP archive
• must have .iemod file extension
• must use compression with the DEFLATE algorithm or no compression
• must have all files and directory names encoded in UTF-8, without BOM
• must not be encrypted
• must not have self-extraction
• must not constitute or contain patched data
• must not span across multiple files
• must not be segmented
• should exclude files and directories listed in the table 'should-exclude'

The folders and files inside IEMOD file:

• must not include names containing characters listed in 'forbidden-characters' table
• must not be listed in case-insensitive 'forbidden-files-directories-all-levels' table
• must not contain names starting with prefixes listed in case-insensitive 'forbidden-files-directories-all-levels-starting-with' table
• must not, in its top-level, contain directories listed in case-insensitive 'forbidden-toplevel-directories' table
• must not, in its top-level, contain files listed in the case-insensitive 'forbidden-toplevel-files' table

Of the files contained within the archive, if WeiDU TP2 file exist, it must be located within a directory that is itself located in the toplevel and it's name must exactly match the name of the directory.

Table forbidden-characters:

<
>
:
"
/
\
|
?
*
\0 # escaped null character

Example:

".:</*Good?|Evil?*\>:."       # literal folder name
".:</*Good?|Evil?*\>:.".tp2   # literal file name

Case-insensitive table forbidden-files-directories-all-levels:

AUX	COM0	COM1	COM2	COM3	COM4	COM5
COM6	COM7	COM8	COM9	CON	CONIN$	CONOUT$
LPT0	LPT1	LPT2	LPT3	LPT4	LPT5	LPT6
LPT7	LPT8	LPT9	NUL	PRN

Example:

CON         # mod data directory
CON\CON.tp2 # WeiDU tp2 file
CON         # weidu executable

Case-insensitive table forbidden-files-directories-all-levels-starting-with:

AUX.	COM0.	COM1.	COM2.	COM3.	COM4.	COM5.
COM6.	COM7.	COM8.	COM9.	CON.	CONIN$.	CONOUT$.
LPT0.	LPT1	LPT2	LPT3	LPT4.	LPT5.	LPT6.
LPT7.	LPT8.	LPT9.	NUL.	PRN.

Example:

AUX.test                # folder name
AUX.test\AUX.test.tp2   # file name

Case-insensitive table forbidden-toplevel-directories:

CD0
CD1
CD2
CD3
CD4
CD5
CD6
cache
characters
data
debugs
dlc
lang
movies
mplayer
mpsave
music
override
portraits
save
script compiler
scripts
sounds
temp
tempsave
workshop

Case-insensitive table forbidden-toplevel-files:

baldur.ini
MConvert.exe
BGConfig.exe
bgmain.exe
bgmain2.exe
charview.exe
chitin.key
dialog.tlk
dialogf.tlk
icewind.exe
icewind.ini
icewind2.exe
idmain.exe
iwd2.exe
torment.exe
torment.ini
weidu.log
weidu.conf

Case-insensitive table should-exclude:

__macosx/
$RECYCLE.BIN/
backup/
.* # all files starting with dot
*.bak
*.iemod
*.temp
*.tmp
Thumbs.db

 

Edited January 4, 2021 by AL|EN