On mod metadata, and INI files

[suy]

First of all, congratulations for the release, of course! I want to start a new thread to not clutter the first one with a very specific issue.

I've noticed form the screenshot that you seem to provide support for mod metadata. While I agree on the value on having a mod metadata file that doesn't require launching WeiDU to obtain it, I find that the choice of INI files is potentially problematic. I imagine the fact that is simple, and that some mods are already parsing it inside WeiDU might be a reason to pick it.

But the loosely specified format can yield different results with different parsers, and I wanted to have your input on the topic.

For example, Jastey's SoD Tweakpack (sorry Jastey, just an example, not to complain on you, on the contrary!) yields me this output when parsing the metadata with a Lua library (and I get the same on C++ with Qt):

21:37 alex@leela /tmp$ lua
Lua 5.2.4  Copyright (C) 1994-2015 Lua.org, PUC-Rio
> inifile = require 'inifile'
> ini = inifile.parse("/home/alex/projects/infinityengine/3rdparty/mods/Jasteys_SoD_Tweakpack/c#sodtweaks/c#sodtweaks.ini")
> print(ini['Metadata']['Description'])
 This tweak pack is mainly meant for the SoD part of BG:EE (except for the last component which introduces Imoen's SoD portrait into BGII). 

Seems fine, right? But Then I get this with Ruby:

irb(main):003:0> ini = IniFile.load('/home/alex/projects/infinityengine/3rdparty/mods/Jasteys_SoD_Tweakpack/c#sodtweaks/c#sodtweaks.ini')
/home/alex/local/gems/gems/inifile-3.0.0/lib/inifile.rb:578:in `error': Could not parse line: "It deals with some tweaks that I found useful for my own game." (IniFile::Error)
        from /home/alex/local/gems/gems/inifile-3.0.0/lib/inifile.rb:532:in `block in parse'
        from /home/alex/local/gems/gems/inifile-3.0.0/lib/inifile.rb:515:in `each_line'
        from /home/alex/local/gems/gems/inifile-3.0.0/lib/inifile.rb:515:in `parse'
        from /home/alex/local/gems/gems/inifile-3.0.0/lib/inifile.rb:400:in `parse'
        from /home/alex/local/gems/gems/inifile-3.0.0/lib/inifile.rb:128:in `block in read'
        from /home/alex/local/gems/gems/inifile-3.0.0/lib/inifile.rb:128:in `open'
        from /home/alex/local/gems/gems/inifile-3.0.0/lib/inifile.rb:128:in `read'
        from /home/alex/local/gems/gems/inifile-3.0.0/lib/inifile.rb:80:in `initialize'
        from /home/alex/local/gems/gems/inifile-3.0.0/lib/inifile.rb:31:in `new'
        from /home/alex/local/gems/gems/inifile-3.0.0/lib/inifile.rb:31:in `load'
        from (irb):3:in `<main>'
        from /usr/lib/ruby/gems/3.0.0/gems/irb-1.3.5/exe/irb:11:in `<top (required)>'
        from /usr/bin/irb:23:in `load'
        from /usr/bin/irb:23:in `<main>'

Woops, what hapenned? The issue is that the line is truncated, and the above that seemingly worked, actually lost a bit of text:

# Short description of the mod, main goals, features etc
Description = This tweak pack is mainly meant for the SoD part of BG:EE (except for the last component which introduces Imoen's SoD portrait into BGII). 
It deals with some tweaks that I found useful for my own game.

So, different implementations can have this quirks...

I don't know how feasible this can be for your implementation or AL|EN's, but ideally, I would propose gradually switching to TOML. This is very, very similar to INIs, so it's easy, but it is well specified, and it supports for example splitting a long string across multiple lines, or having arrays for the installation order. For most mods it would be a matter of copying the INI with the different suffix, adding quotes on the values, and call it a day.

The alternative is of course to narrow the definition of the INI for mod metadata to something much more tight so this doesn't happen. Now that there are at least two implementations (and I've always considered doing one myself, or contributing to Zeitgest, and a few people are doing modding tools like modda or mod_installer), so I think it's no longer a theoretical concern.

 

[argent77]

That's actually the metadata file format that was originally invented for Project Infinity, so I didn't have to reinvent the wheel. There are certainly better formats available, but the INI format is simple enough to be understood by the average modder who doesn't know about yaml, json, and co.

TOML looks promising but it is ultimately @AL|EN's decision whether to make the switch for PI.

[lynx]

The few mods using multiline descriptions just need to be fixed to use a single line description (ok, maybe escaping would work, but it's ugly).

[argent77]

9 hours ago, lynx said:

The few mods using multiline descriptions just need to be fixed to use a single line description (ok, maybe escaping would work, but it's ugly).

Relying on that would work fine in an ideal world. But there will certainly be more mods with multiline descriptions (since this is the natural thing to do if you don't know the exact ini format specs).

I'll try to make the parser more lenient in that regard to show metadata even with malformed description entries.

[suy]

FWIW, seems PI shows just the first line, and I think just the first line is the correct result. The other parsers that I've tried do that as well. I would just clarify so in the wiki page. The desktop file specification can be used for inspiration. It is said that it is delimited by new line characters. I think this makes sense. The format is just not suitable for more.

[argent77]

Without PI's source code available I can only speculate, but I guess that PI performs line-wise parsing of INI content where skipping invalid lines is a trivial matter. WIT uses the more conventional token-based approach which is more flexible but also requires more effort to handle malformed data.

 

29 minutes ago, suy said:

The desktop file specification can be used for inspiration.

It's a good start but the requirement of case-sensitive key names could be problematic. I don't think any of the established INI format specs would cover the PI metadata format precisely.

[suy]

31 minutes ago, argent77 said:

It's a good start but the requirement of case-sensitive key names could be problematic.

Sure, I meant just the idea of how the lines and the keys are specified.

31 minutes ago, argent77 said:

I don't think any of the established INI format specs would cover the PI metadata format precisely.

This is why I worried about the format being potentially problematic... If it's only for PI, then, it either works there, or not, and it doesn't matter. If it is supposed to be for different tools as well, I wish it is something that can work in all of them without having to guess if it's an issue in one tool, the other, or the mod.

There are more tools now. Besides WIT, there are some command line ones, which is unlikely that will read the description, but might want to read the Before and After keys. Before knowing that you would publish your project, I was working on my own a bit on Zeitgeist.

If tool authors are going to ask modders the favor of writing this metadata, better that we do it well. I'm maybe being overcautious, though. It is not a huge incompatibility issue now. I just want to think forward.

[DavidW]

I think “Don’t put line breaks in INI files” is probably an easier lesson to teach people than “use this new format”.

[jastey]

8 hours ago, DavidW said:

I think “Don’t put line breaks in INI files” is probably an easier lesson to teach people than “use this new format”.

This would ahve been a helpful hint even for PI, apparently, back then when I wrote most of the .inis.

The whole discussion would have been great when AL|EN introduced the .inis. I understand where this is coming from, but I am sitting on the other side: having to update 30+ mods with a new format / file will be a great No Thank You from my side.

[suy]

8 hours ago, DavidW said:

I think “Don’t put line breaks in INI files” is probably an easier lesson to teach people than “use this new format”.

Agreed, but if I'm suggesting the new format is because:

1. The new format is 95% the old one. It just requires putting the values in quotes.
2. The new format allows putting multiple lines if the mod author likes so (either type the \n, or put the text in multiple lines with triple quotes instead).
3. The new format has plenty of existing libraries to use. Several to choose from for Java, C# or C++ (for WIT, PI and Zeitgeist respectively). No need for a handwritten parser (though I imagine there should be plenty for INI as well in Java).
4. Less risk of getting bugreports due to incompatibility between apps for this or that quirk.
5. Online validators exist.

I don't think the format itself is a problem. Transitioning to it is surely an inconvenience... That's why I also support keeping the old one. I just don't see such a clear path on how to clarify the format to ensure no future issues arise.

If we go with the old one (which would also make sense, I just want it to be a well informed decision), I can try making a validator for the files, so modders can try it if they want some validation. It still has the risk that somebody else's validator behaves a tiny bit differently.

Thanks.

 

[suy]

10 minutes ago, jastey said:

having to update 30+ mods with a new format / file will be a great No Thank You from my side

Totally understandable! Just to be clear: you would not NEED to change to the new format if both are kept for some time (or forever, with the old being deprecated), similarly to how there are both DESIGNATED and LABEL for mod components. Additionally, this could be done automatically, and even via pull requests so you would only have to press the merge button if you'd liked to.

But again, yes, I understand that it's effort. I just wanted to bring up the possibility of it being worth it. Just that.

[argent77]

10 hours ago, suy said:

Besides WIT, there are some command line ones, which is unlikely that will read the description, but might want to read the Before and After keys.

There also seems to be a way to define component-specific mod order within tp2 scripts by using METADATA tags. An example would be Iwdification.

I don't know if this method is already supported by PI since the above example was the only one I could find. In any case I couldn't find any mentions about it in the PI docs, so I had to guess the format for the most parts.

Edited October 22, 2023 by argent77

[Graion Dilach]

8 minutes ago, argent77 said:

There also seems to be a way to define component-specific mod order within tp2 scripts by using METADATA tags. An example would be Iwdification.

I don't know if this method is already supported by PI since the above example was the only one I could find. In any case I couldn't find any mentions about it in the PI docs, so I had to guess the format for the most parts.

Yes, it was documented in this topic a great while ago. The last posts in this thread are about this one. The way it's developed in PI though that it's a full rewrite of the INI metadata, not an addendum. (and thanks for reminder, I want to update DSotSC-Trilogy's spell components with this.)

Edited October 22, 2023 by Graion Dilach

[argent77]

Good to know, thanks. Looks like I guessed it right so far.

[AL|EN]

I've spent a lot of time When I tried to decide which format I should choose for mod metadata. Way way too much but I guess I can say that I did my homework on that matter. During my research of various configuration file formats such as ini, csv, xml, toml, yaml, HCL, HOCON, json, jsonc, hjson, json5, json6, strictYaml etc the conclusion was: no one wants to deal with file formats created by geeks for geeks just to be able to put mod display name, description and other not-so-important things. No one wants to deal with datatypes even if those are similar to IE datatypes. So it's either ini or custom format. The XML is crap, JSON requires escape sequences, and Yaml requires understanding that spaces matter. I've chosen ini since there are at least some specifications.

Supporting extra file format is not as easy as it seems: Suy want's TOML, Magus want's yaml, someone else wants JSON and I want psd1. JSON and psd1 are built-in inside PowerShell/PI. But for TOML and yaml formats, I would need to write my own parser since it's not possible for PI to use already created parsers because PI is a single-file exe and its update system relies on that. Besides, yaml is unsafe and TOML has already been criticized as 'ini with types' thing. So for now, until critical requirements won't be met, let's stay with ini.