thingy to install mods: modda

[mickabouille]

I've hacked together a util to automate the installation of mods on a IE game (but tests mostly happen on bgX EE variants).

Motivation: I couldn't use PI because I'm not on windows and I was fed up with borking installs because I forgot some mod or some components or failed at ordering. And I don't like clicking things anyway.

This is made to scratch my own itches. It may or may not suit anybody else than me. That would be fine, it fulfilled its purpose (being useful to me), but maybe it can be useful for someone else.

It's mostly a frontend to weidu (needs the executable installed somewhere). You feed it a file like that (that's YAML by the way, forum doesn't highlight that)

global:
  lang_dir: "fr_FR"
  lang_preferences: ["#rx#^fran[cç]ais", french, english, "american english"]
modules:
  - name: DlcMerger
    components: [
      1, # Merge DLC into game -> Merge "Siege of Dragonspear" 
    ]
    location:
      source:
        github_user: Argent77
        repository: A7-DlcMerger
        release: v1.3
        asset: lin-A7-DlcMerger-v1.3.zip
  - name: JanQuest
    components: [0]
    location:
      http: http://akadis.baldursgateworld.fr/Jan/zip/JansQuest1.5.0.zip
  - name: willowisp
    components: # 4 later
      - 0 # Will NPC, shaman stronghold and new shaman kit for BG2EE
      - 1 # Change shaman .tlk string to remove "Ineligible for any stronghold" line
      - 2 # New items for shamans and undead NPCs
      - 3 # Optional: Drider and Dark Treant Enemies
    location:
      path: /home/me/my_mods/static/Will of the Wisp v2.20.zip

Then it will download and unpack (optional), set the language, select and install the components int he given order.

You can't add logic (like install this if this condition but that is this other condition), that's just a recipe that is followed (at its core, it chains weidu invocations).

At some point, this got kind of over-engineered, but that's the most basic way to use it. You can also in theory generate the YAML file from an existing weidu.log but this is not yet tested much.

 

I used it from short installs to test repeated installations of single mod (for translation) or a few mods (to test interactions) to dozens (uh... hundreds) of mods in EET installs. Obviously the longer the list, the longer the install (it can cache the donwloaded archives, but install time dominates for many things, like SCS...).

Main focus is linux (remember, my own itches) on EE, but I took the time to test on windows (EE and original bg2).

Binaries exist for mac but I have no way to know if it does anything at all.

 

Quote

The name sucks

Yeah, I know, my naming sense sucks too.

Quote

Those are binaries, Why should I trust you?

You shouldn't. Download the code, inspect it and build it. Also, stop installing weidu mods because those can do anything too.

 

The project https://github.com/mleduque/modda with some basic usage documentation

The latest release https://github.com/mleduque/modda/releases/tag/v1.3.1 with working binaries for linux, probably working for win64 and no idea for macOS

[marchitek]

I like idea of such package manager approach. I don't think it is overengineered.

Suggestions from my side:

• support for labels
• generation of yaml snippet from tp2 with all components included (easier to edit then write it from scratch)
• support for subcomponents and maybe READLN (unless it exists, maybe I missed that)

[mickabouille]

22 minutes ago, marchitek said:

generation of yaml snippet from tp2 with all components included (easier to edit then write it from scratch)

At some point, I (briefly) considered using a parser for tp2 files. That is there was a formal syntax I could use or translate into a usable grammar for me, but I couldn't find it in the weidu source  (or maybe I did but couldn't make head or tails with it).

Actually finding components in a .tp2 file can be tricky. You can do maybe ^\s*BEGIN\s+@\d+ but then you don't see non traified component names (so you add it to your regex) and then you grab some commented-out components...

And then you have to actually find what @1 in BEGIN @1 stands for in another file (which one? ).

The best you can do is have a grammar for tp2 and .tra (but the .tra part is far easier)

The second best is... ask weidu --list-components-json (AFAIK this one isn't even in the weidu documentation). That's what modda --list-components does, but then I had forgotten I even wrote that! That _could_ be the base for the generation.

But then you need to decide how you handle GROUPs. And REQUIRE_PREDICATEs too.

Of course you can just add all components and hand edit incompatibilities... That could be an idea actually.

23 minutes ago, marchitek said:

support for labels

I'm not sure I understand what those are.

32 minutes ago, marchitek said:

subcomponents

This... would be tricky because of the lack of internal information on the mod

24 minutes ago, marchitek said:

READLN (unless it exists, maybe I missed that)

Yes, that would be useful. I thought about doing it, but had other priorities, then added automatic patching, so I pachted ACTION_READLNs out, then I didn't needed it as much and sort of forgot about it.

I'd rather modders stop using then but I guess they exist and everybody must live with them.

But _this one_ could be possible if working a bit like expect (the tool)

 

[mickabouille]

Ah I see now, the generation of the yaml file is the actual problem. I'm so used to it that I didn't see it anymore.

I usually start with a trick

global:
  lang_dir: "fr_FR"
  lang_preferences: ["#rx#^fran[cç]ais", french, english, "american english"]
modules:
  - name: bg1ub
    components: ask # woops I don't know what components are there!

Then when modda falls on theis "components: ask" it will stop itself and you find yourself with a "normal" weidu prompt (that is actually the real weidu prompt).

I asks you the exact same questions you would have when executing setup-whatever.exe and installs everything normally.

I would then `modda reverse` the weidu.log file into a yaml file... or just copy/paste (in block mode)

 

EDIT: not saying it couldn't be streamlined... or automated.

Edited April 15, 2023 by mickabouille

[MikeX]

Thanks for the tool. I'm playing with it on windows.

For the 'reverse' command, could you also read the component names from weidu.log and put them into the yaml file?

~TOBEX/TOBEX.TP2~ #0 #100 // TobEx - Core: v28
~TOBEX/TOBEX.TP2~ #0 #101 // Apply Concentration Check On Damage [WIP]: v28
~TOBEX/TOBEX.TP2~ #0 #102 // Awaken On Damage: v28
~TOBEX/TOBEX.TP2~ #0 #103 // Blindness As Spell Description: v28

...

 

[mickabouille]

You can try

modda reverse --export-component-name true --output example.yml

I could not extract the names as comments (`#`) but it inserts them like

- name: faiths_and_powers
  components:
  - index: 25
    component_name: 'Choosee a Sphere System -> nuFnP: a new sphere system (fewer spheres, more balanced, closer to PnP): 0.85sd19'
  - index: 31
    component_name: 'Install Cleric kits: 0.85sd19'
  - index: 33
    component_name: 'Install Druid kits: 0.85sd19'
  - index: 36
    component_name: 'Install Paladin kits -> ...for all those, PLUS Neutral/Evil/demihuman deities: 0.85sd19'
  - index: 45
    component_name: 'Revised Paladin Spell Table: 0.85sd19'
  - index: 75
    component_name: 'Alter Priests'' Weapon/Armor Usability: 0.85sd19'
  - index: 80
    component_name: 'Apply Sphere System (REQUIRED for the sphere system to work): 0.85sd19'
  - index: 91
    component_name: 'Multiclass Druids: 0.85sd19'
  - index: 92
    component_name: 'Multiclass Shamans: 0.85sd19'
  - index: 95
    component_name: 'Multiclass Cleric kits (install AFTER all other kit mods!!): 0.85sd19'
  - index: 99
    component_name: 'Apply FnP Multiclass Kits to NPCs: 0.85sd19'

The name is ignored at the install time of course, so it's not that different from

- name: faiths_and_powers
  components:
  - 25 # Choosee a Sphere System -> nuFnP: a new sphere system (fewer spheres, more balanced, closer to PnP): 0.85sd19

 

[mickabouille]

In general

$ modda
Weidu install automation

Usage: modda <COMMAND>

Commands:
  install          install mods
  search           Search all module declarations in the manifest with the given name
  list-components  List the available components of a weidu mod (by index)
  invalidate       Remove a downloaded mod from the cache
  reverse          generate a skeleton manifest YAML file from a `weidu.log` file
  help             Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version
$ modda help reverse
generate a skeleton manifest YAML file from a `weidu.log` file

Usage: modda reverse [OPTIONS] --output <OUTPUT>

Options:
  -o, --output <OUTPUT>
          Name of the file that will be generated
  -l, --export-language <EXPORT_LANGUAGE>
          If set, the `language` field in mod definitions will be generated (default: `false`) [possible values: true, false]
  -c, --export-component-name <EXPORT_COMPONENT_NAME>
          If set, the component names will be generated (default: `true`) [possible values: true, false]
  -h, --help
          Print help

 

[MikeX]

16 hours ago, mickabouille said:

You can try

modda reverse --export-component-name true --output example.yml

That's what I did, but did not work:

Part of my WeiDU.log:

~TOBEX/TOBEX.TP2~ #0 #100 // TobEx - Core: v28
~TOBEX/TOBEX.TP2~ #0 #101 // Apply Concentration Check On Damage [WIP]: v28
~TOBEX/TOBEX.TP2~ #0 #102 // Awaken On Damage: v28
~TOBEX/TOBEX.TP2~ #0 #103 // Blindness As Spell Description: v28

Part of the resulting yaml:

- name: tobex
  language: 0
  components:
  - 100
  - index: 101
    component_name: '101'
  - index: 102
    component_name: '102'
  - index: 103
    component_name: '103'
  - index: 105
    component_name: '105'

I used latest release v1.3.1.

16 hours ago, mickabouille said:

In general

There seems to be a difference about the options that are showing with --help on my version...

Edited April 16, 2023 by MikeX

[marchitek]

2 hours ago, mickabouille said:

And then you have to actually find what @1 in BEGIN @1 stands for in another file (which one? ).

The best you can do is have a grammar for tp2 and .tra (but the .tra part is far easier)

The second best is... ask weidu --list-components-json (AFAIK this one isn't even in the weidu documentation). That's what modda --list-components does, but then I had forgotten I even wrote that! That _could_ be the base for the generation.

But then you need to decide how you handle GROUPs. And REQUIRE_PREDICATEs too.

Sure. It gets more complicated if you would like to handle everything, but one thing is tool to check dependencies between components, to check install order etc and other is tool for installation automation. If you are doing this second thing, you can relay on people to prepare correct YAML and your tool would fetch packages nad install it according to specification.

What I had in mind was just something to generate modda YAML with all possible components for given mod and then I can manually delete those that I don't want in my installation etc. Components name are not required here (although I can agree if it would generate comments like in your example it would be super useful).

 

2 hours ago, mickabouille said:

3 hours ago, marchitek said:

support for labels

I'm not sure I understand what those are.

https://weidu.org/~thebigg/README-WeiDU.html#hevea_default212

You can use it to resolve component id: https://weidu.org/~thebigg/README-WeiDU.html#hevea_default557

[Lurker]

4 hours ago, mickabouille said:

At some point, this got kind of over-engineered, but that's the most basic way to use it. You can also in theory generate the YAML file from an existing weidu.log but this is not yet tested much.

I don't want to derail, but perhaps somebody finds the following useful.

I'm kind of in the same boat (Linux+Wine), and was thinking on doing something comparable, but on a much, much smaller/simpler scale (just a bash script), especially without builtin download functionallity. I think the greatest pain is the time needed if one uses WeiDU interactively. And I'm on EET, so some added complexity, more to process, and I've got a rather old CPU.

My current workflow (more like "running the gauntlet") is something like this:

• I have a "bg2ee.eet-core" directory as baseline (with just the base EET, dlcmerged-BG1+BG1UB, and HQ_SOUNDCLIPS_BG2EE)
• I have a prepared .tgz with all mods I plan to apply, and a download directory with the individual mods, prefixed with "nn_" for install order reference, or later use a WeiDU.log from a previous install
• I rsync the "bg2ee.eet-core" to a new WIP directory and also extract the "mod.tgz" into it
• semi-manually apply some mods (--force-install--list [component1] [component...])
• use "rsync -av --compare-dest=/full/path/to/bg2ee.eet-core/ bg2ee-wip/ diff/[named_step]/" to have a differential "snapshot" directory
• at arbitrary points create a .tgz from such a differential snapshot directory for archiving/"rollbacking"
• to roll back the bg2ee-wip, "rsync -av --delete-before bg2ee.eet-core/ bg2ee-wip/" and then  e.g. "rsync -av diff/55_after_quests/ bg2ee-wip/"
• last step is always "Generalized Biffing -> Biff all files", and I
  • keep a snapshot.tgz from directly before biffing
  • create a complete .tgz of bg2ee-wip/ after biffing
• I usually include the output of `date -I` in the name of the final.tgz and the respective mod.tgz

I didn't want to go too much into detail, so a bit of abstraction was necessary in the steps above. Hope it's comprehensible.

As a side note, having multiple working directories can also save time when e. g. testing how some tweaks work together, or if my manually mutilated mod works as intended, because I can run multiple installation processes (and even game instances) in parallel.

All in all maybe not overly sophisticated, but since I'm happy on the command line and badly abuse my bash history, the manual labor part isn't overly painful. The time waiting for WeiDU to install mods often is.

To automate the installation, I'd create a bash script that just calls WeiDU as (often as) needed, with parameters for "mod name", "--language 0" (almost always english, else ) and "--force-install--list [components]". That script maybe doesn't even need to be parameterised, just change the script as needed. Less complexity.

Going into overengineering territory a bit, a bash script that parses a WeiDU.log to generate said script from the last paragraph as a template for easy editing should also be doable.

Eh, this may come off as a rant or an empty promise (which I definitely didn't do!), but perhaps posting this publically motivates me enough to actually putting money where my mouth is. Sometime.

[mickabouille]

18 minutes ago, marchitek said:

What I had in mind was just something to generate modda YAML with all possible components for given mod and then I can manually delete those that I don't want in my installation etc.

THAT would actually be possible (and not that hard)

[mickabouille]

15 minutes ago, Lurker said:

As a side note, having multiple working directories can also save time when e. g. testing how some tweaks work together, or if my manually mutilated mod works as intended, because I can run multiple installation processes (and even game instances) in parallel.

rsync snapshot is a cool idea, I hadn't thought about it.

I use another small (much simpler) tool to create symbolic link farms from an immutable single installation https://github.com/mleduque/iedup that I could have included in modda... if I thought it could be portable (this one has no release, so no binaries atm).

So my process is actually as short as

iedup [...] && modda install -m [...]

At one point I tried to imagine a checkpointing  workflow based on git. I expected it to explode because of the binary files, but git was far more robust than I though (I didn't push the idea though, but I think I just I would have had to define a .gitignore carefully if mods didn't write BACKUP data inside themselves )

15 minutes ago, Lurker said:

To automate the installation, I'd create a bash script that just calls WeiDU as (often as) needed, with parameters for "mod name", "--language 0" (almost always english, else ) and "--force-install--list [components]".

That sums up what modda does. Well the core part (it also downloads the mods if you tell it where to find them) (and patches them if you ask it to).

 

15 minutes ago, Lurker said:

To automate the installation, I'd create a bash scrip

That's how it (modda) started, actually. That is until I acknowledged (for the nth time) that I couldn't write bash scripts at all. And  that I never enjoyed it when I had to. So I stop bashing myself hurting myself with bash

Edited April 15, 2023 by mickabouille

[Lurker]

9 minutes ago, mickabouille said:

rsync snapshot is a cool idea, I hadn't thought about it.

I use another small (much simpler) tool to create symbolic link farms from an immutable single installation https://github.com/mleduque/iedup that I could have included in modda... if I thought it could be portable.

I was thinking about using links myself in some way, perhaps even with rsync, but coudn't figure it out before my patience ran out. The solution with --compare-dest turned out to be good enough, especially compared to always making a full .tgz, which I did before. Ouch.

The catch is I don't know how to do an "incremental delete", that's why I bite the bullet and do a full .tgz after biffing. Still good enough.

9 minutes ago, mickabouille said:

At one point I tried to imagine a checkpointing  workflow based on git. I expected it to explode because of the binary files, but git was far more robust than I though (I didn't push the idea though, but I think I just I would have had to define a .gitignore carefully if mods didn't write BACKUP data inside themselves )

I haven't got a clue about using git. The last time I used a VCS, Subversion was still a thing.

But since the dawn of 2023, when I got back to BG, I'm slowly getting used to browsing github.

9 minutes ago, mickabouille said:

That's how it (modda) started, actually. That is until I acknowledged (for the nth time) that I couldn't write bash scripts at all. And  that I never enjoyed it when I had to. So I stop bashing myself hurting myself with bash

It's the other way around for me. I never bothered learning anything else, as long as I either got the job done with bash, or until I decided that it's not worth to bother at all...

Making that stuff robust enough to be used by other people is something else. Way too much responsibility...

I wish I was as quick to (re-)learn things as I am to unlearn them when being idle.

[Endarire]

I used NPM (Node Package Manager for JavaScript) and I appreciate the approach you've done here, even though I'm on Windows.

[subtledoctor]

8 hours ago, mickabouille said:

Yeah, I know, my naming sense sucks too.

It’s still somehow better than mine!