Some more CLI helpers for automated installations on Linux

[Lurker]

Although mickabouille definitely beat me to it, I nevertheless hacked together two simple scripts to ease the installation of a larger number of mods.

Since right know I'm really tired but can't sleep, I'll try to make this brief, because foggy brain because really tired. And as it turns out, I also got really rusty in writing bash, so please bear with me.

My personal use case: I'm on Linux and I use Wine. So no Project Infinity. I'm used to getting mods the old fashioned way by manual downloading and dumping them into the game directory. This usually results in several files like "setup-<modname>.exe" and their respective directories.

wmodinstall.bash sources another file, that mostly should contain a mod name plus a number of components per line, and tries to install these.

wparse.bash takes an existing WeiDU.log and converts it into a file format (dumped at stdout) that wmodinstall.bash can source. The output should be curated before as needed, obviously. Still better than having to type all that shit in manually.

The functionality of both scripts is pretty basic, and while I tried to catch some most(?) of the stuff that some people (like myself) tend to mess up, there's obviously still some room to do really stupid stuff. It's a crutch triwalker(*), not a Swiss army knife.

I put some comments in, but cannot really estimate how self-explanatory it is. I'll create an empty reply after this posting, so that I can put the code there into code tags, if desired for changelog/update information. Won't start using git just because of two simple shell scripts (see bottom of this comment).  Although none of the available syntax highlighters are really fitting, at least they add some color.

Feedback is appreciated.

There's a rooster roaring outside. I hate that critter.

(*) As the feeling of accomplishment for creating something that does exactly what it is supposed to do settles in, I decided that this metaphor deserved an upgrade. And I'm fully aware that my method of self-advertising really sucks. 

 

Download links: (the most recent version is currently determined by the "latest update:" tag of this topic and the changelog.)

• wmodinstall-native.bash
  • preferred method; expects linux weidu and weinstall present and executable somewhere in $PATH
• wmodinstall.bash
  • initial method; using windows binaries for installation; slow
• wparse.bash
  • generates installable "payload" data from an existing WeiDU.log
• wmodinstall.payload
  • example "payload"
• WeiDU.log
  • corresponding to above "payload"

 

KNOWN ISSUE(S):

• "debug" option only applies to "core" functionality, since the payload is simply sourced by bash
  •  any shell commands in the payload file will be executed every time the installer script is invoked
• Mods containing whitespace in their name aren't accounted for
  • so far, the only mod that I know of with where this applies is Black Hearts, which can be easily fixed on the mod's side
• sloppy reset of IFS might be an issue

Edited November 2, 2023 by Lurker
modified known issue(s); restructured inital comment

[Lurker]

Changes (2023-05-02):

wparse.bash

1. added workaround/support for Oversight

Changes (2023-04-26):

wmodinstall-native.bash

1. initial version
2. minimal-effort switch to native weinstall, major performance increase

Changes (2023-04-23):

wmodinstall.bash

1. will now also tolerate chitin.key in all uppercase

Changes (2023-04-20):

wmodinstall.bash

1. added option to ignore warnings
2. added WeiDU parameters --skip-at-view --no-exit-pause
3. added references to g3
4. tried to improve general quality

wparse.bash

1. no major functional change
2. see points 3. and 4. above

Edited May 2, 2023 by Lurker

[mickabouille]

Ah I see you've stumbled upon d0questpack too

And I see you use the mod-provided setup-XXX.exe. I was about to say you can use a native weidu executable on a wine game (I did sometimes), but that may be dependent on a filesystem that is case-preserving. Wine will nullify this problem.

 

And separating log with the timestamp is a nice idea. It allows to have components of a mod install at different place but still get logs.

[Lurker]

1 hour ago, mickabouille said:

Ah I see you've stumbled upon d0questpack too

Yep. Ugly caused ugly.

1 hour ago, mickabouille said:

And I see you use the mod-provided setup-XXX.exe. I was about to say you can use a native weidu executable on a wine game (I did sometimes), but that may be dependent on a filesystem that is case-preserving. Wine will nullify this problem.

Old habits die hard. Just recently started replacing all those setup-*-exe with symlinks to a single weidu.exe. The Wine being easy on case sensitivity thing was a nice learning experience, indeed.

1 hour ago, mickabouille said:

And separating log with the timestamp is a nice idea. It allows to have components of a mod install at different place but still get logs.

Thanks.

cdtweaks I think had 4 different invocations in my last setup, so I was getting tired of having to remember to rename the setup log after each step. I like to keep these around.

Any thoughts on my "parser"?

EDIT: Sorry, the not-up-to-date?-download counter confused me. Noticing the d0questpack stuff would be quite a feat without looking at the parser first. I put the "" around parser because I'm still not sure if it really qualifies to be called as such. Well, the script doesn't care about its own name in the end.

Edited April 20, 2023 by Lurker

[mickabouille]

I must admit my lack of proficiency with awk but I think I recognized that resemble what I do (though my version would probably call for sed as I use a regexp).

That's a good thing the file format is very simple. But I would rather the charset was specified. As it is, each mod will write in its own charset, and you can end up with a mix of utf8, cp12XX and, I guess, other encodings.

[Lurker]

Damn, I asked for feedback. That's not something that I had on my radar at all. I don't think that's something I want to address from within the script.

Have to say that I'm quite happy with the current result.

[mickabouille]

I think you should be safe (maybe?).

I don't expect awk to be encoding aware, or not in a way that would matter here. Apart from delimiters, it should process bytes without caring about the contents in bytes, I expect.

As more me, i handled is the in the crudest manner : `from_utf8_lossy`, meaning I won't crash or abort, but the text in the "component name" might be garbled and/or contain replacement characters..

I don't even feel bad for it as there is (probably) no way to correctly read a file with a mix of encoding (and no marker).

[Lurker]

The way I use awk here is also the only way I know how to use awk, although it's a bit more useful/powerful when dealing with multiple columns at once. I try to stick with bash builtins as much as possible, but ease-of-use, lazyness and readability are valid factors to include external helpers. The performance hit with so many calls to external applications inside the while loop is very noticable, can't deny that.

My recent discovery of

VAR="BLAbla"
echo ${VAR^} ${VAR^^} ${VAR,} ${VAR,,}
      BLAbla  BLABLA   bLAbla  blabla

VAR="blaBLA"
echo ${VAR^} ${VAR^^} ${VAR,} ${VAR,,}
      BlaBLA  BLABLA   blaBLA  blabla

filled me with joy.

Edited April 20, 2023 by Lurker

[Lurker]

I've added an example curated payload file and the resulting WeiDU.log to the topic. And since the resulting WeiDU.log is identical to the one previously sourced by the parser, I can happily claim that this stuff really works.

I used the "nowarn" option and answered the remaining interactive prompts manually.

My starting point is an archived EET-Core installation. That's why the payload actually begins with Ascension.

This setup takes over 8 hours to complete on my machine (Xeon E3-1245 v3, comparable to i7-4770).

I've never tried the Linux version of WeiDU, and would be interested to know if the native version works significantly faster?

Edited April 25, 2023 by Lurker

[mickabouille]

A small (mostly) unrelated remark on your example payload

modinstall a7-convenienteenpcs "3 203 204 205 500 661" 0
#^^^^^^^^^ 661: needs ACTION_READLN patch or manual input
[...]
modinstall npc_ee "2000" 0

with (convenient ee npc) 661 being  Change class/kit: Hexxat : Choose your class (readln)

(npc_ee) 2000 is NPC class/stat//kit choice which adds the tome to every joignable character and (AFAIK) allows nearly every change (exceptions are dual class characters, that need component 1200 to make then non-dual, and triple classes who can't choose a kit)

Hexxat is single class already. I don't know what class/kit change you needed for Hexxat (begin ACTION_READLN it can't appear in weidu.log) but the NPC_EE tome should handle this case, shouldn't it ? Without the inconvenience of readln.

EDIT: wow! Hexxat already has 10 options in convenient (excepting the action_readln one, including thief)!

Edited April 22, 2023 by mickabouille

[mickabouille]

While I'm at this

#^^^^^^^^^ 3183: needs ACTION_READLN patch or manual input

This one you can bypass with cdtweaks.txt I think

EDIT which doesn't help as you need to patch the cdtweaks.txt file anyway...

Edited April 22, 2023 by mickabouille

[Lurker]

My premise when building this particular setup was to have a "stable baseline", that would be fine without relying on EEKeeper for "convenience tweaks". I consider NPC_EE as an in-game replacement for EEKeeper. But,  aside from some very half-assed tests for basic functionality, I have no experience with how well NPC_EE works in a "serious" playthrough. So it's a win-win that is has to be installed so very late.

If I relied on NPC_EE alone, I would/could also have omitted the other NPC-related mods.

29 minutes ago, mickabouille said:

(npc_ee) 2000 is NPC class/stat//kit choice which adds the tome to every joignable character and (AFAIK) allows nearly every change (exceptions are dual class characters, that need component 1200 to make then non-dual, and triple classes who can't choose a kit)

I'm aware of the dual class caveat. For Imoen, it doesn't apply, because if I use her, I already have full control over her development right out of Candlekeep. This is EET, after all. For Anomen, I dont' want to mess with his first class anyway, because of npckit. And Nalia I turned into an Adventurer with npckit, which boosts her open locks and find traps a bit.

29 minutes ago, mickabouille said:

Hexxat is single class already. I don't know what class/kit change you needed for Hexxat (begin ACTION_READLN it can't appear in weidu.log) but the NPC_EE tome should handle this case, shouldn't it ? Without the inconvenience of readln.

EDIT: wow! Hexxat already has 10 options in convenient (excepting the action_readln one, including thief)!

I suffered though SoD (as part of EET, but without further content changes) only once, and that wording should make it obvious how well I liked it. The only Beamdog NPC that I actually like and would have around for a whole saga run is Baeloth(*). Hexxat, no thanks. Neither for her personality, nor for her class(**). But when the choice popped up, I thought "What would make me at least think about the possibility to take her, maybe when hell froze over?" So I changed her class to Sorcerer, TRUE CLASS. A very fitting drop-in for Baeloth, at least mechanically.

In my current game, I EEKeepered Aerie into a Sorcerer, with Baeloth's MR. She has almost no kills, but she ensures that my Archer charname is a one-elf-show, with the remaining companions only for flavor or to block doors. Oh, almost forgot, Haer'Dalis as an invisible Skald (<-- Rogue Rebalancing) is also very nice.

(*) Both for his class and his voice acting - gives Edwin a run for his money. And he's completely un-romanceable, so less cringe.

(**) In my current game, my Archer got a "roleplay epiphany" and destroyed her in her tomb on the spot. A fitting place, and the game also ensures that you have to close the door on your way out.

[mickabouille]

That was mostly the combination of both (661, with the added inconvenience of ACTION_READLN _plus_ 2000 which makes 661 a no-op) that seemed unnecessary

But of course if you're still unsure about npc_ee working as claimed... Better make sure.

 

[Lurker]

It's a display of my goodwill. Or that I'm an idiot. Spending extra effort on the perhaps universally most disliked NPC, instead of just disabling it.

Otoh, shame on Argent77 for not including the, without any doubt, most reasonable choice.

Edited April 22, 2023 by Lurker

[Lurker]

Minor convenience update for wmodinstall.bash.

Edited April 23, 2023 by Lurker