Alternatives to WeiDU? Or, how to write a smarter sound set installer?

[subtledoctor]

5 hours ago, Taylan said:

Tell me that's not a lot nicer!

Not quite a fair comparison - with the Weidu .tp2 code given by Kjeron above, you can move some arrays and such out to simple text files (as you have done in your hypothetical python version), and hide the DEFINE_ACTION_FUNCTION away in a functions library (as you have done) and then the actual Weidu code need only be about 10 lines. 

I’ve leaned this wacky scripting mod installer language, so it seems sensible to me, while even the short & simple version of your python code looks like gibberish; you are experienced with various nice & modern languages, and the idiosyncratic BG2 modding language looks weird and unworkable. 

Not to say Weidu isn’t weird and idiosyncratic. Just that, apart from offloading certain kinds of compute-heavy tasks for which Weidu is inefficient to lua or perl or whatever... on balance I don’t see the benefit of reinventing the wheel. 

Note, Weidu has evolved steadily to accomplish a lot of pretty specific goals: parsing .2da tables, editing .d/.bcs files, bit-editing game resource files with particular structures, etc. etc. You might define python libraries to handle soundset additions, but that is an exceedingly limited use-case. Care to do the same for all the other ways people mod the games? Not that you couldn’t... but you’d go crazy!

[Graion Dilach]

@TaylanThe pseudo-code is mostly right. Mostly because it ignores the UI lua edit to switch the CHARSND.2DA coloumn header into a tlk entry and give the player a human-readable input instead of the WAV name (keep in mind that IE is limited to the 8.3 filename format) and to fall back to -1 on the entries which doesn't exist because the game is able to reroll the sound variant where it's supported so you don't have to duplicate sounds just to fill the variants (was a nice assumption though). How proper Python it is, I've no idea though (personal opinion; Python itself is ugly compared to C/C++/C#) and honestly I agree with @subtledoctor here, reinventing the wheel because "it's not a general-purpose language" does sound like a lot of wasted effort for covering the sheer amount of things WeiDU accomplished.

Funnily I'm coming from a modding community where the games are modded through INIs and I tell you - INI is severely limited, because it offers little grouping (sure this isn't required for soundsets but for the rest it will). C&C Generals devs broke the INI format for their own gain back in 2002-2003 and when OpenRA started, it also ran into issues with INI's limitations and they went with a YAML subset in the end instead (which is relevant because YAML would work better as a general-purpose "decompiled" variant of some of the file formats here). Your example also ignores the fact that you need to extract CHARSND.2da first from the game (something which WeiDU covers, haven't looked up iesh to see if that does as well).

In my topic, there are two people already who have posted custom soundsets using my mods as template. I can understand that if you only want to touch soundsets, then WeiDU is a huge burden to deal with and offers more complications than bonuses in this regard, but I don't think that is enough to justify reinventing a wheel.

[Taylan]

11 hours ago, subtledoctor said:

Not quite a fair comparison - with the Weidu .tp2 code given by Kjeron above, you can move some arrays and such out to simple text files (as you have done in your hypothetical python version), and hide the DEFINE_ACTION_FUNCTION away in a functions library (as you have done) and then the actual Weidu code need only be about 10 lines.

You're missing a crucial aspect of the code I've written: it allows you to define *everything* about the sound sets in INI files. The Python code doesn't ever need to be touched. I.e. a person creating a sound set package wouldn't need to know either Python nor TP2, just put WAV files in a directory and write a definitions.ini file, then put the installer.py script in the same directory and zip it up.

And while you may be used to TP2 and not Python, I have to disagree that finding one cleaner over the other is purely a matter of taste and/or prior experience. I've been programming professionally for about 10 years (+2 before that) and worked with a variety of languages including some weird ones: sh, C, Objective-C, C#, Java, Scheme, PHP, JavaScript, Python, and XSLT. Usually learning new languages is quite easy at that point. But I have difficulty figuring out what TP2 code does, even with the documentation at my side, because it's really idiosyncratic. On the other hand I've only ever written about 2 serious Python scripts in my life and was able to produce straightforward code that mostly does the right thing (not completely but only because I don't know IE modding details), is probably easy to understand for anyone with a bit of programming experience, and allows the person creating a sound set package to not touch a single line of code.

Apparently WeiDU is written in OCaml, so I wonder whether that might have affected the way the TP2 language works. But I doubt it, because from what I know OCaml is a very clean language that supports "functional programming" style. (A bit like Scheme which I'm a near-expert in, but with different syntax, static typing, and built-in pattern matching... I could probably learn enough of it in 1-2 hours to be able to make changes to WeiDU's code. )

I think anyone who's seriously worked with a proper programming language would agree that TP2 makes some things, that should be simple, unnecessarily difficult.

7 hours ago, Graion Dilach said:

@TaylanThe pseudo-code is mostly right. Mostly because it ignores the UI lua edit to switch the CHARSND.2DA coloumn header into a tlk entry and give the player a human-readable input instead of the WAV name (keep in mind that IE is limited to the 8.3 filename format) and to fall back to -1 on the entries which doesn't exist because the game is able to reroll the sound variant where it's supported so you don't have to duplicate sounds just to fill the variants (was a nice assumption though). How proper Python it is, I've no idea though (personal opinion; Python itself is ugly compared to C/C++/C#) and honestly I agree with @subtledoctor here, reinventing the wheel because "it's not a general-purpose language" does sound like a lot of wasted effort for covering the sheer amount of things WeiDU accomplished.

Funnily I'm coming from a modding community where the games are modded through INIs and I tell you - INI is severely limited, because it offers little grouping (sure this isn't required for soundsets but for the rest it will). C&C Generals devs broke the INI format for their own gain back in 2002-2003 and when OpenRA started, it also ran into issues with INI's limitations and they went with a YAML subset in the end instead (which is relevant because YAML would work better as a general-purpose "decompiled" variant of some of the file formats here). Your example also ignores the fact that you need to extract CHARSND.2da first from the game (something which WeiDU covers, haven't looked up iesh to see if that does as well).

Indeed I forgot the fallback to -1, gotta fix that. I guess I could try adding the UI thing too.

About extracting CHARSND from the game: my idea was that the imaginary "weidu" library I used in the Python code would take care of that. I.e. the .parse2DA function would search for it in the game directory including overrides.

I definitely wouldn't propose to use INI for anything more complicated. I just thought, "how could we make the creation of sound set packages as easy as can be," and using INI for the definitions was the first thing to pop to mind. You could just as easily use Python for the definition file. I just did a little Googling and it seems it would be just as simple, e.g. a "definitions.py" file could look like this, allowing the sound set package author to add some special code there if needed for some reason:

Spoiler

name = 'MyCustomSoundSet'
language = 'en_us'

sounds = {
  'greeting1': 'greeting1.wav',
  'greeting2': 'greeting2.wav',
  'greeting3': 'greeting2.wav',
  'battlecry1': 'battlecry1.wav',
  'battlecry2': 'battlecry1.wav',
  'battlecry3': 'battlecry2.wav',
  ...
}

subs = {
  'greeting1.wav': 'Yes?',
  'greeting2.wav': 'What is it?',
  'battlecry1.wav': 'You will fall by my hand!',
  'battlecry2.wav': 'No mercy for enemies!',
  ...
}

 

 

[subtledoctor]

12 hours ago, Taylan said:

And while you may be used to TP2 and not Python, I have to disagree that finding one cleaner over the other is purely a matter of taste and/or prior experience. 

I didn’t say that it is purely a matter of taste, or that they are in some way equal but for the user’s experience. My point was (and it should not be read as some kind if logical conclusion, but more a supposition or prediction) that using a better language to reproduce everything Weidu does will require way more effort than you would ultimately gain from the efficiencies gained therefrom. I could be wrong! But if I had to put money down, it would be on “just stick with Weidu.” 

12 hours ago, Taylan said:

But I have difficulty figuring out what TP2 code does, even with the documentation at my side, because it's really idiosyncratic.

To be fair, it might also be because the documentation is awful. I can’t tell you how many times I’ve known what I wanted to do, read and seemingly understood the documentation, and yet could not create working syntax. The language is idiosyncratic, and the documentation does not fully identify or teach the idiosyncrasies. 

Edited June 26, 2021 by subtledoctor

[Jarno Mikkola]

1 hour ago, subtledoctor said:

I didn’t say that it is purely a matter of taste, or that they are in some way equal but for the user’s experience. My point was ... that using a better language to reproduce everything Weidu does will require way more effort than you would ultimately gain from the efficiencies gained therefrom.

You know guys, you could summon @Wisp to comment about this, and he has to use a OCaml ... or whjat the cheese it ever was to program Weidu... so, I would say that you are just using a ready made library to expand the resources you already have at your disposal. What utility, well, let me introduce you to the Command promp. Yes, the magic.

 

1 hour ago, subtledoctor said:

The language is idiosyncratic, and the documentation does not fully identify or teach the idiosyncrasies. 

You know, you could rewrite any and all the tutorials the weidu readme has about the documentation, and have your cake and eat it too... but I have yet to find too many tutorials by you. In fact I have seen only a few links from you to any weidu code, and so I ... suggest you to do more of that, rather than to do tutorials, at first.

[Ardanis]

16 hours ago, Taylan said:

You're missing a crucial aspect of the code I've written: it allows you to define *everything* about the sound sets in INI files. The Python code doesn't ever need to be touched. I.e. a person creating a sound set package wouldn't need to know either Python nor TP2, just put WAV files in a directory and write a definitions.ini file, then put the installer.py script in the same directory and zip it up.

I think you're the one missing it, honestly. There's nothing stopping you from writing code with exact same functionality in WeiDU syntax, put it into useful_functions_lib.tph, and call it with modder-defined arguments whenever you need it. That's what more advanced mods have been doing for over a decade.

You could say that your version is more compact because it doesn't need to ship WeiDU.exe along, but as far as I'm aware Python isn't natively supported by Windows either.

PS Speaking of Python, it's an awful dysfunctional language that breaks apart if you don't meticulously keep your tabs and whitespaces in order every time you update a tiny bit of your code.

Edited June 26, 2021 by Ardanis

[Jarno Mikkola]

16 minutes ago, Ardanis said:

PS Speaking of Python, it's an awful dysfunctional language that breaks apart if you don't meticulously keep your tabs and whitespaces in order every time you update a tiny bit of your code.

Are you telling me that there isn't a program in the world that can help to auto manage this.

[Taylan]

18 minutes ago, Ardanis said:

I think you're the one missing it, honestly. There's nothing stopping you from writing code with exact same functionality in WeiDU syntax, put it into useful_functions_lib.tph, and call it with modder-defined arguments whenever you need it. That's what more advanced mods have been doing for over a decade.

I know it's probably possible. It seems the language can read arbitrary files and process their contents in a (probably) Turing complete way. However, in programming language lingo we have this term called a "Turing tarpit." Just because something is possible in a language, doesn't mean that language is as good as any other language. The most extreme examples would be languages like Brainfuck and Unlambda which are designed specifically to be so minimal that writing any useful software in them is practically impossible, even if technically possible. TP2 is obviously not that extreme, but I do honestly find the idea of using it as a general-purpose language pretty horrible.

Keeping track of spaces and tabs in Python should really not be an issue for anyone who uses a halfway decent editor. Even Notepad++ should do. In fact that's what I'm planning to use for this pet project... so I'll be off to creating a proof-of-concept Python library and sound set package installer then, wish me luck.

[Ardanis]

39 minutes ago, Taylan said:

However, in programming language lingo we have this term called a "Turing tarpit." Just because something is possible in a language, doesn't mean that language is as good as any other language.

True, but there's also saying "don't fix what's not broken". WeiDU's syntax is certainly far from optimal, not in the least because it started as a simple dialog updater/compiler and over two decades grew into what it is now, but it gets the job done and is widely accepted by the IE community as the standard of IE modding.
I.e. you can certainly do things your own way, but expect a lot complaints if it won't be fully compatible with WeiDU-based install routine

Edited June 26, 2021 by Ardanis

[grodrigues]

6 hours ago, Ardanis said:

Speaking of Python, it's an awful dysfunctional language that breaks apart if you don't meticulously keep your tabs and whitespaces in order every time you update a tiny bit of your code.

I have used Python since the days of 2.2 and never had, or even have heard of anyone having, the problems you speak of.

A Python library for modding would be so much better than WeiDU that it is not even funny. The problem is that, to even have a chance of being adopted and replace WeiDU (as opposed to being used for some one-off project or other) it would have to lure modders with 100% WeiDU compatibility (meaning it could install and uninstall WeiDU mods and WeiDU would be able to uninstall anything installed by it) and then on top offer some really, really good advantages for the non-programming savvy. It is not enough to have something that is better; it must be much better so that the effort in learning pays off. For example, I have written enough of such a library that you can write something like:

for p in Path(".").glob("*.itm"):
    if p.is_file():
        with Item.patch(p) as f:
            if f.type == itemtypes.longsword:
                f.min_strength = 10

This is, minus some details with working with in-game files that I have elided, a direct translation of the example in the READ_BYTE and PATCH_IF WeiDU tutorial. There are no offsets to remember, error checking is much stricter, you have to try *really* hard to put the item in an invalid state, etc. The syntax is sane, there are no wacky loony things like dynamic scoping to trip you up, you have actual data structures and a vast, solid library of code to reuse, etc. But is it enough? No, not really.

20 years of an established code base amounts to an awful lot of inertia.

[CamDawg]

28 minutes ago, grodrigues said:

A Python library for modding would be so much better than WeiDU that it is not even funny.

Sure, and the same could be said for C/C#/C++, Java, Perl*, hell, even OCaml, etc. It's the formal programming structures and libraries (that WeiDU can kinda, sorta do) that we're longing for.

28 minutes ago, grodrigues said:

The problem is that, to even have a chance of being adopted and replace WeiDU (as opposed to being used for some one-off project or other) it would have to lure modders with 100% WeiDU compatibility (meaning it could install and uninstall WeiDU mods and WeiDU would be able to uninstall anything installed by it) and then on top offer some really, really good advantages for the non-programming savvy.

Yep.

* But seriously, don't use Perl**.
** Like, for real.

[Jarno Mikkola]

45 minutes ago, grodrigues said:

and WeiDU would be able to uninstall anything installed by it)

Why the F would Weidu need to be able to uninstall it's stuff at all ? Cause if the other thing is possible to do 100%, then this is completely unneeded. Like seriously. Cause if it takes a .tp2 file and is able to do everything in it without any questions, and without weidu(.exe), then weidu becomes irrelevant to it. But I bet that's a hard job to do, so sure at first this would be a one off ... at first.

[Ardanis]

3 hours ago, grodrigues said:

This is, minus some details with working with in-game files that I have elided, a direct translation of the example in the READ_BYTE and PATCH_IF WeiDU tutorial. There are no offsets to remember, error checking is much stricter, you have to try *really* hard to put the item in an invalid state, etc. The syntax is sane, there are no wacky loony things like dynamic scoping to trip you up, you have actual data structures and a vast, solid library of code to reuse, etc. But is it enough? No, not really.

The only reason there's no such consolidated data library for WeiDU is because nobody bothered yet. There're numerous functions - either included in WeiDU or shipped with mods - for adding/editing file contents without offset juggling.

Tutorial is rather outdated anyway, as it only shows the basics. The more advanced mods have been operating their data on higher levels for over a decade.

WeiDU syntax certainly has its share of issues, the LONG_NAMES_IN_CAPS eyesore being one of my pet peeves since ever, but lack of ability to handle your data on abstract level is not one of those.

Edited June 27, 2021 by Ardanis

[Magus]

13 hours ago, grodrigues said:

for p in Path(".").glob("*.itm"):
    if p.is_file():
        with Item.patch(p) as f:
            if f.type == itemtypes.longsword:
                f.min_strength = 10

 

For each Python snippet I will match and raise you a Weidu one.

COPY_EXISTING_REGEXP GLOB ~*.itm~ ~override~
  LPF GET_ITEM_TYPE RET type
  PATCH_IF type == ITM_TYPE_LONGSWORD
  LPF ALTER_ITEM_REQS INT_VAR min_str=10

You have to write the underlying code in both cases. So this argument is null and void.

The real advantage of a general purpose language is that it's... general purpose. It has much larger community, better docs, more features, isn't dependent on 1-2 mantainers.

And, it's actually a useful skill to have. While weidu is hardly applicable anywhere else.

12 hours ago, CamDawg said:

Sure, and the same could be said for C/C#/C++, Java, Perl*, hell, even OCaml, etc. It's the formal programming structures and libraries (that WeiDU can kinda, sorta do) that we're longing for.

Not really. A C++ lib would only be developed by a chosen few. Learning curve is far too steep. Python, on the other hand, is widely regarded as one of the best choices for beginners, and since most modders are, and will be on that level, this makes it a very good choice if you want the ecosystem to grow naturally (modders developing their own functions and modules as needed).

10 hours ago, Ardanis said:

WeiDU syntax certainly has its share of issues, the LONG_NAMES_IN_CAPS eyesore being one of my pet peeves since ever, but lack of ability to handle your data on abstract level is not one of those.

Oh, that one you can easily enough work around with a preprocessor. I entertained some thoughts on that, but nothing concrete yet.

18 hours ago, Taylan said:

so I'll be off to creating a proof-of-concept Python library and sound set package installer then, wish me luck.

A "soundset installer" is useless. The only thing that can have a future is a weidu installation/uninstallation/rollback compatible generic installer. If I had to guess, I'd say a week or two should be enough to put together a PoC (assuming that you start with iesh). So, ball's in your court. I'll be watching this with interest.

Edited June 27, 2021 by Magus

[Taylan]

Happy to see more people in this thread who agree to my point of view.

Yes, I suppose WeiDU compatibility would be crucial if such a project is to become something serious, and a sound set installer alone wouldn't get too much attention, especially since the WeiDU-based one seems to do the job just fine even if it isn't as squeaky clean. I frankly haven't written much code yet after all, but found myself thinking a lot about how exactly the project could look if it is to become serious. The current idea looks like this:

PieDU

A Python library & modding framework for IE games that aims to be dirt easy to install, clean in design, and easy to create mods for.

The name is of course a homage to WeiDU, and the "Pie" part sounds like "Py" which is what a lot of Python tools/libs/etc. like to put in their name. It could have been called "PeiDU" to be closer to WeiDU, but I've come up with the neat backronym "Python Infinity Engine Doctoring Utility" which I like. Also, pie is yummy. So PieDU it is.

PieDU is installed by unpacking it into a stand-alone sub-directory within an IE game installation folder. This means you have to install it once for each IE game you want to use it for, but that's not a problem because it mainly consists of a bunch of Python source files which are small in size. (I expect that it would still be under 1MB after it can do most things you'd want it to do.) It doesn't matter what you call the sub-directory for PieDU, but as a convention let's go with "PieDU" (duh). The following graph shows a sample IE game installation folder together with a PieDU installation:

e.g. C:\Program Files\Steam\steamapps\common\Baldur's Gate II Enhanced Edition
|- data
|- lang
|- Baldur.exe
|- ... etc ...
|- PieDU
   |- start.py (launcher script to start the PieDU UI)
   |- README.txt (always good to have documentation!)
   |- piedu (the actual PieDU Python library, don't touch)
   |- mods (drop your mods into here; the following are examples)
      |- SoundSet-NWNF5
      |- SoundSet-Blah
      |- SwordCoastStratagems
      |- TweaksAnthology

It seems that Python is trivial to install on Windows. There's official installers on the homepage (the kind where you just click Next a few times) as well as a free "app" in the Microsoft Store if you wanna go that route. Either way, once Python is installed, you can simply double-click on any blah.py file to run it, so all you need to do to start PieDU is to double-click on start.py. You can of course also start it from the command-line like "python start.py" on any OS.

The user-interface could simply be console-based (don't have to do anything special for this; double-clicking on a blah.py automatically starts the Python program in a console), or it could use a GUI at some point in the future, since Python also has a built-in cross-platform GUI library. Whenever you start PieDU, it would ask you what you want to do, like install/uninstall any of the mods registered with PieDU.

To register mods with PieDU you just drop them into the mods directory. They should be self-contained sub-directories like shown above. They would contain all the mod files plus a definitions.txt file that declares a bit of meta-data about the mod, like a pretty name, some documentation, and maybe a "mod type."

A mod type could for instance be "sound set" which tells PieDU to invoke a predefined sound set installer procedure (like the one I've written previously in this thread) to handle the installation of the mod, so the mod author doesn't need to write any Python code. Other mod types could be "class kit" or "npc" for instance. And of course there would be a fall-back mod type called "custom" where you can add any Python code you want.

Some trivial WeiDU "compatibility" would at first be offered by taking the easiest possible way: just allow people to drop a WeiDU mod into the mods directory, write a trivial definitions.txt that says it's a mod of type "weidu" and says where the main .exe is (or even just auto-detect all this). Then when the user starts PieDU and selects that mod for installation, PieDU just starts the mod's WeiDU installer.

Thoughts/criticism/feedback?

To be honest I fear that I'll end up being all talk and little action because my actual job is keeping my quite busy and I have some other hobby projects too, but let's see where it goes.