mickabouille

On 4/21/2023 at 3:18 PM, Lurker said:

This setup takes over 8 hours to complete on my machine (Xeon E3-1245, comparable to i7-4770).

I've never tried the Linux version of WeiDU, and would be interested to know if the native version works significantly faster?

@Lurker

https://cpu.userbenchmark.com/Compare/Intel-Xeon-E3-1245-V2-vs-AMD-Ryzen-7-4800H/m5622vsm1032976 but then the CPU is only one part of the whole, I'm guessing the disk IO counts there too (I'm using a loopback on a SDD for what it's worth).

TLDR for the same install as yours (some versions may be different) but using my installer

• total install time using linux native weidu: 21 minutes
• installation for ascension (the first mod) using a bash script that runs wine weidu.exe: 28 minutes

You asked  the question, I got curious and then spent far too much time on this

Some call that nerd-sniping

1 hour ago, AL|EN said:

yaml is most difficult configuration file format for average Joe,

Yes, most definitely not very accessible.

I *think* you can feed json instead of yaml, but it's not much better, it's still mostly a dev thing. Probably a bit more accessible but definitely more verbose.

You can feed the program a weidu.log to generate a yaml file too, but that means you already made some install prototype, or you are confortable with writing a weidu.log by hand which... is not entirely impossible.

I could certainly devise a simple file format for the simplest use case (like one mod name per line, nothing more), have the user answer what weidu prompts and then spits the resulting selection in a file.

A gui... may be a bit out of scope (as I see it).

So many ideas...Now I feel I didn't think things through.. which I did not in fact, that was a purely iterative succession of what ifs.

8 hours ago, lynx said:

The answer to "how to deal with potential interactivity" has always been to use good old expect.

That... was my first inclination too. And it still is the most likely direction if I ever try to tackle this (ATM, I crudely paper over it and patch interactive mods to not be interactive).

10 hours ago, AL|EN said:

Is that mean that even if PI would exist for Linux, you would still prefer command line?

It's become hard to answer now.

I think when I didn't know the first thing about mods, weidu, components and all that, I would have found a GUI very valuable anyway.

I would probably have ended up trying to hand-modify whatever textual representation PI would have had, if any.

Now... as I said, I made a program just to my tastes (which are certainly _not_ everybody's tastes) so the comparison would be unfair.

But yes, I'm biaised toward text files anyway (maybe because I'm a software developer).

On 1/28/2023 at 4:02 PM, mickabouille said:

Hi, following you recent release, I updated the french translation there https://github.com/subtledoctor/d5_Random_Tweaks/pull/10

• The new strings you added in english were translated
• The ones you added had some slight fixes done
• I replaced the component names (which go in weidu.log) with pure ASCII (using UTF-8 for these was an error on my part).

(Really, shoudln't weidu normalize the content of weidu.log in a single charset?)

Hi @subtledoctor

Might I get you interested in having a look at this translation update?

Would you consider that a bug in weidu?

While it's fresh in your head,maybe you could attempt to describe the behavior at https://github.com/WeiDUorg/weidu/issues ?

I guess the project is not extremely active and advances by fit and starts, with a massive backlog to boot.

But even having an issue filled could serve as documentation of sorts for the issue?

That was mostly the combination of both (661, with the added inconvenience of ACTION_READLN _plus_ 2000 which makes 661 a no-op) that seemed unnecessary

But of course if you're still unsure about npc_ee working as claimed... Better make sure.

While I'm at this

#^^^^^^^^^ 3183: needs ACTION_READLN patch or manual input

This one you can bypass with cdtweaks.txt I think

EDIT which doesn't help as you need to patch the cdtweaks.txt file anyway...

A small (mostly) unrelated remark on your example payload

modinstall a7-convenienteenpcs "3 203 204 205 500 661" 0
#^^^^^^^^^ 661: needs ACTION_READLN patch or manual input
[...]
modinstall npc_ee "2000" 0

with (convenient ee npc) 661 being  Change class/kit: Hexxat : Choose your class (readln)

(npc_ee) 2000 is NPC class/stat//kit choice which adds the tome to every joignable character and (AFAIK) allows nearly every change (exceptions are dual class characters, that need component 1200 to make then non-dual, and triple classes who can't choose a kit)

Hexxat is single class already. I don't know what class/kit change you needed for Hexxat (begin ACTION_READLN it can't appear in weidu.log) but the NPC_EE tome should handle this case, shouldn't it ? Without the inconvenience of readln.

EDIT: wow! Hexxat already has 10 options in convenient (excepting the action_readln one, including thief)!

I think you should be safe (maybe?).

I don't expect awk to be encoding aware, or not in a way that would matter here. Apart from delimiters, it should process bytes without caring about the contents in bytes, I expect.

As more me, i handled is the in the crudest manner : `from_utf8_lossy`, meaning I won't crash or abort, but the text in the "component name" might be garbled and/or contain replacement characters..

I don't even feel bad for it as there is (probably) no way to correctly read a file with a mix of encoding (and no marker).

24 minutes ago, Lurker said:

Embarrassingly, this came to mind only very recently: weidu.exe --help

That works too, but I like the searchable web page

I must admit my lack of proficiency with awk but I think I recognized that resemble what I do (though my version would probably call for sed as I use a regexp).

That's a good thing the file format is very simple. But I would rather the charset was specified. As it is, each mod will write in its own charset, and you can end up with a mix of utf8, cp12XX and, I guess, other encodings.

1 hour ago, Lurker said:

I'm inclined to guess that this isn't the most up-to-date documentation for WeiDU...?

I got frustrated I couldn't find one more recent and started using the one inside the release zip https://github.com/WeiDUorg/weidu/releases/tag/v249.00 which should be up to date (probably).

1 hour ago, Lurker said:

Although I'm still hesitating to add more parameters to my scripts. getopts isn't the most accessible feature in bash...

Be careful or you'll end up with 4500 lines of code

Ah I see you've stumbled upon d0questpack too

And I see you use the mod-provided setup-XXX.exe. I was about to say you can use a native weidu executable on a wine game (I did sometimes), but that may be dependent on a filesystem that is case-preserving. Wine will nullify this problem.

And separating log with the timestamp is a nice idea. It allows to have components of a mod install at different place but still get logs.

On 4/19/2023 at 4:58 AM, Lurker said:

How do you catch deal with WeiDU warnings? Is there a (hidden?) option in WeiDU to suppress them, or to change the exit code? (<-- stupid question)

That something I don't like much about weidu, it sets a non-zero exit code on warnings (oK, minor inconvenience, just check if this was a warning or an error code) and doesn't allow to decide if a warning is innocuous or breaks everyrthing (probably not doable weidu side anyway).

I chose to be pedantic by default and not let warnings pass, leaving the user to check the actual warning and `ignore_warnings` on the "component group". I even tried to find which component warned but it was fragile (parsing setup-XXX.debug oooof!) so I dropped that.

Now I just group the warning and non-warning  components separately.

I'd love weidu to have different exit codes for different types of warnings. Some can usually be ignored (no effect changed or something like that).

And like you, I'd like a weidu flag to ignore warnings.

On 4/19/2023 at 4:58 AM, Lurker said:

Better question: Is there a way to prevent the "Press ENTER to exit." prompt after  INSTALLED WITH WARNINGS (WeiDU exit code 3)?

I think that `--no-exit-pause`

I also add `--skip-at-view` because some mods open the readme in a text editor at the end (sometimes multiple times, for each component!).

On 4/19/2023 at 4:58 AM, Lurker said:

I'm also looking for a (simple) way to prevent interruption of an install sequence.

Whether it's warnings or errors that interrupt your installation, you can check the exit code of weidu and decide not to interrupt.

I think with bash you'd need to not have `set -e`, I'm not sure of the details.

For my tool you just

modda install -m my-install.yml --no-stop-on-warn

to ignore warnings. i didn't create the option to ignore errors. It would be as easy but I didnd't see the point. Maybe I'll do it.

I very rarely use it anyway, I prefer to whitelist component by component after checking them, but I've found it useful on some throwaway installs made to check things in some mods.

39 minutes ago, MikeX said:

Ok, v1.4.0 works, thank you very much.

Sorry about that

16 hours ago, marchitek said:

What I had in mind was just something to generate modda YAML with all possible components for given mod and then I can manually delete those that I don't want in my installation etc

This one should work now  (v1.4.0)

modda help append-mod

though forget about the language option, copy-paste bad!

15 minutes ago, Lurker said:

As a side note, having multiple working directories can also save time when e. g. testing how some tweaks work together, or if my manually mutilated mod works as intended, because I can run multiple installation processes (and even game instances) in parallel.

rsync snapshot is a cool idea, I hadn't thought about it.

I use another small (much simpler) tool to create symbolic link farms from an immutable single installation https://github.com/mleduque/iedup that I could have included in modda... if I thought it could be portable (this one has no release, so no binaries atm).

So my process is actually as short as

iedup [...] && modda install -m [...]

At one point I tried to imagine a checkpointing  workflow based on git. I expected it to explode because of the binary files, but git was far more robust than I though (I didn't push the idea though, but I think I just I would have had to define a .gitignore carefully if mods didn't write BACKUP data inside themselves )

15 minutes ago, Lurker said:

To automate the installation, I'd create a bash script that just calls WeiDU as (often as) needed, with parameters for "mod name", "--language 0" (almost always english, else ) and "--force-install--list [components]".

That sums up what modda does. Well the core part (it also downloads the mods if you tell it where to find them) (and patches them if you ask it to).

15 minutes ago, Lurker said:

To automate the installation, I'd create a bash scrip

That's how it (modda) started, actually. That is until I acknowledged (for the nth time) that I couldn't write bash scripts at all. And  that I never enjoyed it when I had to. So I stop bashing myself hurting myself with bash

18 minutes ago, marchitek said:

What I had in mind was just something to generate modda YAML with all possible components for given mod and then I can manually delete those that I don't want in my installation etc.

THAT would actually be possible (and not that hard)

In general

$ modda
Weidu install automation

Usage: modda <COMMAND>

Commands:
  install          install mods
  search           Search all module declarations in the manifest with the given name
  list-components  List the available components of a weidu mod (by index)
  invalidate       Remove a downloaded mod from the cache
  reverse          generate a skeleton manifest YAML file from a `weidu.log` file
  help             Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version
$ modda help reverse
generate a skeleton manifest YAML file from a `weidu.log` file

Usage: modda reverse [OPTIONS] --output <OUTPUT>

Options:
  -o, --output <OUTPUT>
          Name of the file that will be generated
  -l, --export-language <EXPORT_LANGUAGE>
          If set, the `language` field in mod definitions will be generated (default: `false`) [possible values: true, false]
  -c, --export-component-name <EXPORT_COMPONENT_NAME>
          If set, the component names will be generated (default: `true`) [possible values: true, false]
  -h, --help
          Print help

You can try

modda reverse --export-component-name true --output example.yml

I could not extract the names as comments (`#`) but it inserts them like

- name: faiths_and_powers
  components:
  - index: 25
    component_name: 'Choosee a Sphere System -> nuFnP: a new sphere system (fewer spheres, more balanced, closer to PnP): 0.85sd19'
  - index: 31
    component_name: 'Install Cleric kits: 0.85sd19'
  - index: 33
    component_name: 'Install Druid kits: 0.85sd19'
  - index: 36
    component_name: 'Install Paladin kits -> ...for all those, PLUS Neutral/Evil/demihuman deities: 0.85sd19'
  - index: 45
    component_name: 'Revised Paladin Spell Table: 0.85sd19'
  - index: 75
    component_name: 'Alter Priests'' Weapon/Armor Usability: 0.85sd19'
  - index: 80
    component_name: 'Apply Sphere System (REQUIRED for the sphere system to work): 0.85sd19'
  - index: 91
    component_name: 'Multiclass Druids: 0.85sd19'
  - index: 92
    component_name: 'Multiclass Shamans: 0.85sd19'
  - index: 95
    component_name: 'Multiclass Cleric kits (install AFTER all other kit mods!!): 0.85sd19'
  - index: 99
    component_name: 'Apply FnP Multiclass Kits to NPCs: 0.85sd19'

The name is ignored at the install time of course, so it's not that different from

- name: faiths_and_powers
  components:
  - 25 # Choosee a Sphere System -> nuFnP: a new sphere system (fewer spheres, more balanced, closer to PnP): 0.85sd19

Ah I see now, the generation of the yaml file is the actual problem. I'm so used to it that I didn't see it anymore.

I usually start with a trick

global:
  lang_dir: "fr_FR"
  lang_preferences: ["#rx#^fran[cç]ais", french, english, "american english"]
modules:
  - name: bg1ub
    components: ask # woops I don't know what components are there!

Then when modda falls on theis "components: ask" it will stop itself and you find yourself with a "normal" weidu prompt (that is actually the real weidu prompt).

I asks you the exact same questions you would have when executing setup-whatever.exe and installs everything normally.

I would then `modda reverse` the weidu.log file into a yaml file... or just copy/paste (in block mode)

EDIT: not saying it couldn't be streamlined... or automated.

22 minutes ago, marchitek said:

generation of yaml snippet from tp2 with all components included (easier to edit then write it from scratch)

At some point, I (briefly) considered using a parser for tp2 files. That is there was a formal syntax I could use or translate into a usable grammar for me, but I couldn't find it in the weidu source  (or maybe I did but couldn't make head or tails with it).

Actually finding components in a .tp2 file can be tricky. You can do maybe ^\s*BEGIN\s+@\d+ but then you don't see non traified component names (so you add it to your regex) and then you grab some commented-out components...

And then you have to actually find what @1 in BEGIN @1 stands for in another file (which one? ).

The best you can do is have a grammar for tp2 and .tra (but the .tra part is far easier)

The second best is... ask weidu --list-components-json (AFAIK this one isn't even in the weidu documentation). That's what modda --list-components does, but then I had forgotten I even wrote that! That _could_ be the base for the generation.

But then you need to decide how you handle GROUPs. And REQUIRE_PREDICATEs too.

Of course you can just add all components and hand edit incompatibilities... That could be an idea actually.

23 minutes ago, marchitek said:

support for labels

I'm not sure I understand what those are.

32 minutes ago, marchitek said:

subcomponents

This... would be tricky because of the lack of internal information on the mod

24 minutes ago, marchitek said:

READLN (unless it exists, maybe I missed that)

Yes, that would be useful. I thought about doing it, but had other priorities, then added automatic patching, so I pachted ACTION_READLNs out, then I didn't needed it as much and sort of forgot about it.

I'd rather modders stop using then but I guess they exist and everybody must live with them.

But _this one_ could be possible if working a bit like expect (the tool)

I've hacked together a util to automate the installation of mods on a IE game (but tests mostly happen on bgX EE variants).

Motivation: I couldn't use PI because I'm not on windows and I was fed up with borking installs because I forgot some mod or some components or failed at ordering. And I don't like clicking things anyway.

This is made to scratch my own itches. It may or may not suit anybody else than me. That would be fine, it fulfilled its purpose (being useful to me), but maybe it can be useful for someone else.

It's mostly a frontend to weidu (needs the executable installed somewhere). You feed it a file like that (that's YAML by the way, forum doesn't highlight that)

global:
  lang_dir: "fr_FR"
  lang_preferences: ["#rx#^fran[cç]ais", french, english, "american english"]
modules:
  - name: DlcMerger
    components: [
      1, # Merge DLC into game -> Merge "Siege of Dragonspear" 
    ]
    location:
      source:
        github_user: Argent77
        repository: A7-DlcMerger
        release: v1.3
        asset: lin-A7-DlcMerger-v1.3.zip
  - name: JanQuest
    components: [0]
    location:
      http: http://akadis.baldursgateworld.fr/Jan/zip/JansQuest1.5.0.zip
  - name: willowisp
    components: # 4 later
      - 0 # Will NPC, shaman stronghold and new shaman kit for BG2EE
      - 1 # Change shaman .tlk string to remove "Ineligible for any stronghold" line
      - 2 # New items for shamans and undead NPCs
      - 3 # Optional: Drider and Dark Treant Enemies
    location:
      path: /home/me/my_mods/static/Will of the Wisp v2.20.zip

Then it will download and unpack (optional), set the language, select and install the components int he given order.

You can't add logic (like install this if this condition but that is this other condition), that's just a recipe that is followed (at its core, it chains weidu invocations).

At some point, this got kind of over-engineered, but that's the most basic way to use it. You can also in theory generate the YAML file from an existing weidu.log but this is not yet tested much.

I used it from short installs to test repeated installations of single mod (for translation) or a few mods (to test interactions) to dozens (uh... hundreds) of mods in EET installs. Obviously the longer the list, the longer the install (it can cache the donwloaded archives, but install time dominates for many things, like SCS...).

Main focus is linux (remember, my own itches) on EE, but I took the time to test on windows (EE and original bg2).

Binaries exist for mac but I have no way to know if it does anything at all.

Quote

The name sucks

Yeah, I know, my naming sense sucks too.

Quote

Those are binaries, Why should I trust you?

You shouldn't. Download the code, inspect it and build it. Also, stop installing weidu mods because those can do anything too.

The project https://github.com/mleduque/modda with some basic usage documentation

The latest release https://github.com/mleduque/modda/releases/tag/v1.3.1 with working binaries for linux, probably working for win64 and no idea for macOS

38 minutes ago, MikeX said:

Switch sign of Armor Class Bonus by multiplying it with -1, then it's more intuitive (since lower numbers are better) to calculate the resulting armor class

I actually prefer having "3 point bonus to AC" rather than +3 or -3. Bonus means "better" and malus means "worse" in any case.

For absolute values, it's a bit more complicated...