Some more CLI helpers for automated installations on Linux

[mickabouille]

On 4/21/2023 at 3:18 PM, Lurker said:

This setup takes over 8 hours to complete on my machine (Xeon E3-1245, comparable to i7-4770).

I've never tried the Linux version of WeiDU, and would be interested to know if the native version works significantly faster?

@Lurker

https://cpu.userbenchmark.com/Compare/Intel-Xeon-E3-1245-V2-vs-AMD-Ryzen-7-4800H/m5622vsm1032976 but then the CPU is only one part of the whole, I'm guessing the disk IO counts there too (I'm using a loopback on a SDD for what it's worth).

TLDR for the same install as yours (some versions may be different) but using my installer

• total install time using linux native weidu: 21 minutes
• installation for ascension (the first mod) using a bash script that runs wine weidu.exe: 28 minutes

 

You asked  the question, I got curious and then spent far too much time on this

Some call that nerd-sniping

Edited April 25, 2023 by mickabouille

[Lurker]

Thanks, I added the missing "v3" to the posting you quoted. Not significantly different. Game is on a SATA SSD, local ext4, writeback and 16 GB RAM, so shouldn't matter.

I was suspecting that I might shoot myself in the foot for staying with Wine for WeiDU, but that's a dramatic difference. Like WTF?!?

I called for it. You sniped me (back).

[Lurker]

Ugh. I'm clearly not in the mood to learn the intricacies of the native tools right now. tolower? Disk images? Special file system mount options?

Having argued against vendor lock-in in the other thread just before, I need to admit how comfy the golden cage of Wine feels right now.

 

 

Edited April 25, 2023 by Lurker

[mickabouille]

It's still running in the background, 1:54 bst is installed now.

The first 2 (ascension and iwdification) are beasts and were long to install, then there is a plethora of smaller ones.

But then we'll come to stratagems... which is long even with the native one (it took many dozens of minutes on an install I did last year but... there were quite a lot more mods in this one (100+) and I think SCS parse each resource multiple times)

 

EDIT:

interrupted at npckit because of missing iconv (I took the lin-***.zip version).

real    127m8,662s
user    0m6,915s
sys     0m6,016s

I thin I won't go on further

Edited April 25, 2023 by mickabouille

[Lurker]

1 hour ago, mickabouille said:

TLDR for the same install as yours (some versions may be different) but using my installer

• total install time using linux native weidu: 21 minutes
• installation for ascension (the first mod) using a bash script that runs wine weidu.exe: 28 minutes

Just to be sure: Your tool installs a payload comparable to mine in 21 minutes, whereas my solution takes 8+ hours?

That's really hard to grasp.

16 minutes ago, mickabouille said:

It's still running in the background, 1:54 bst is installed now.

The first 2 (ascension and iwdification) are beasts and were long to install, then there is a plethora of smaller ones.

But then we'll come to stratagems... which is long even with the native one (it took many dozens of minutes on an install I did last year but... there were quite a lot more mods in this one (100+) and I think SCS parse each resource multiple times)

I don't get a cohesive impression from all of this right now...?

[Lurker]

Grmph. Since I can't get (native) weidu or weinstall to do anything but spitting out nonsensical error messages about supposedly missing files, a bit of waiting for a working result isn't so bad.

EDIT: Okay, so just to enable the ability to actually USE the native binaries to cope with the "creative" way this environment + EE "improvements" deal with FiLeNaMeS, in my case (Debian Bullseye, ext4), I'd need to:

• upgrade to a kernel with CONFIG_UNICODE=y from backports
• enable the "casefold" feature for the file system and remount
  • does that even work on an existing fs?
• chattr +F on an empty directory and then dump the game files into it

Yeah. Sure.

Edited April 25, 2023 by Lurker

[mickabouille]

3 hours ago, Lurker said:

Your tool installs a payload comparable to mine in 21 minutes, whereas my solution takes 8+ hours?

I don't think that's my tool vs your. I'd think yours should be somewhat faster than mine (straight invocation with no logic), though maybe not really perceptible.

It's more native weidu vs weidu run with wine I think.

3 hours ago, Lurker said:

I don't get a cohesive impression from all of this right now...?

The point was, the install starts straight with two extremely long mods (ascension, iwdification), that each took more than 20 minutes, but after that there were smaller mods so the mean time per mod got smaller and smaller. The rest is blabbering.

 

3 hours ago, Lurker said:

EDIT: Okay, so just to enable the ability to actually USE the native binaries to cope with [...]

Yeah, I remember debian not activating casefolding in the kernel. The maintainer had no use for it so it wasn't useful for anybody...

There are other FS with casfolding, actually I think ext4/casefolding is not the most used here, but I didn't try any other option.

I'm of no help there, sorry...

[Lurker]

1 hour ago, mickabouille said:

It's more native weidu vs weidu run with wine I think.

I didn't mean to imply anything else, though "yours vs. mine" applies simply by yours doing native and mine going through Wine.

1 hour ago, mickabouille said:

The point was, the install starts straight with two extremely long mods (ascension, iwdification), that each took more than 20 minutes, but after that there were smaller mods so the mean time per mod got smaller and smaller. The rest is blabbering.

I'm still flabbergasted by the huge difference in runtime and if I understand your metrics correctly. So 21 minutes start-to-finish for a whole payload with native weidu, but anything thereafter you reported was slow-boating the same payload with Wine-weidu? I know from experience that some mods install exceptionally slow, I guess due to inefficient libraries.

1 hour ago, mickabouille said:

Yeah, I remember debian not activating casefolding in the kernel. The maintainer had no use for it so it wasn't useful for anybody...

Yes, but it's reported as implemented since late 5.13.9, while the userland is already there in Bullseye (defaults to 5.10). I was able to create a fs with the feature enabled, but then couldn't mount it.

1 hour ago, mickabouille said:

There are other FS with casfolding, actually I think ext4/casefolding is not the most used here, but I didn't try any other option.

Will test it in the future, I guess. It fits the "going native" approach in the end, whereas an approach like described here wasn't in the least compelling to go native instead of just sticking with Wine.

So, in the end it seems that migrating my script to weinstall could be less effort than adapting the work environment.

1 hour ago, mickabouille said:

I'm of no help there, sorry...

No matter, thanks anyway.

Edited April 25, 2023 by Lurker

[Lurker]

Timed installation, full auto, payload from topic minus the three interactive components, Wine+weidu.exe:

real    446m3,296s
user    0m2,207s
sys     0m3,681s

As above, but with a loop-mounted JFS (case-insensitive support enabled) on top of ext4, native weidu/weinstall:

real    23m52,318s
user    20m8,322s
sys     3m39,623s

Disregarding the line terminators, both installations end with identical WeiDU.logs.

Speedup factor ~18.
 

Edited April 26, 2023 by Lurker

[mickabouille]

Some stranger and stranger ideas go through my head, like having a normal directory (case sensitive) exported locally with samba...

[Lurker]

Attached new wmodinstall-native.bash to topic. weidu/weinstall has to be present somewhere in $PATH.
 

[Lurker]

1 hour ago, mickabouille said:

Some stranger and stranger ideas go through my head, like having a normal directory (case sensitive) exported locally with samba...

Apropos stranger thoughts: Maybe if weidu on Linux was more popular, it would be better adapted to actually deal with the idiosyncrasies of the environment it's supposed to work in...?

Edited April 26, 2023 by Lurker

[Lurker]

Hmm, I noticed that the string tables differ slightly in size and number of entries (in Near Infinity). The Wine setup has 319124 entries, the native setup 319130.

Generally, how much reproducibility can be expected from repeated builds with WeiDU on an identical baseline?

EDIT: To answer my own question somewhat, a binary diff between two consecutive setups using the same recipe and environment is annoyingly huge. But the resulting dialog.tlk are identical.

Also annoying, native weidu ignores umask and creates files with all bits set for ugo.

Edited April 26, 2023 by Lurker

[Lurker]

Updated wparse.bash.

[Lurker]

On 4/26/2023 at 2:25 PM, mickabouille said:

Some stranger and stranger ideas go through my head, like having a normal directory (case sensitive) exported locally with samba...

@mickabouille I always had a soft spot for XFS, but could never justify using that instead of ext4 for my usual use cases. Maybe this might be it?

XFS supports ASCII only case-insensitive filename lookup and Copy-on-write (CoW). Using "cp --reflink" then should  be a nice and very effective way for dealing with test-installs.

Alternatively, ext4 + casefolding for Unicode support, and OverlayFS to add CoW.

Okay, now I'll have to agree that your idea regarding Samba is really strange.

Edited May 3, 2023 by Lurker