Alternatives to WeiDU? Or, how to write a smarter sound set installer?

[subtledoctor]

7 hours ago, Taylan said:

just allow people to drop a WeiDU mod into the mods directory, write a trivial definitions.txt that says it's a mod of type "weidu" and says where the main .exe is (or even just auto-detect all this). Then when the user starts PieDU and selects that mod for installation, PieDU just starts the mod's WeiDU installer.

Don’t even need the mod’s .exe file. Every mod’s executable file is, after all, no more or less than a renamed copy if weidu.exe. So just ship weidu.exe within PieDU, search for any valid .tp2 files in the game folder, and install a selected mod via command-line arguments. (Hey, this is starting to sound familiar.) 

Which raises the question, why must the mod design/installer be tied to a GUI? Why not leave things simple in PieDU, and let stuff like Project Infinity or MWL invoke Weidu mods and Python mods, as each is chosen by the user? (Not that I would turn my nose up at a cross-platform IE mod-installing GUI!)

[AL|EN]

@Taylan

When there is a chance for something new I take closer look at the topic with attention and curiosity. A novice can shed new light on old, stiff and unchanged things for years. This gives a chance that at least some aspects of modding can be improved. Even if a lite bit.

Looking at your soundset installer project, I see potential in it. I would suggest focusing on it because it does not require a lot of work, it is independent of other tools and will allow you to make mistakes (you will do them for sure), learn from them, so that you will know how to design bigger things if you ever get on with them.

But let's focus on your PieDU proposal:
The natural course of things is that as a developer, the first thing you do is look at the programming language, code, directory structure and size of files. This makes you set the project on failure from the very beginning. What you should focus on are: players, modders, mod ecosystem. Then, on existing solutions, tools. Then on how everything connects and what can be improved.

Your current design inherits all of WeiDU's current fundamental flaws. It's too much to write elaborates here. If you want to chat you can find me on Discord: https://discord.com/invite/yTzjMTb

10 hours ago, Taylan said:

To be honest I fear that I'll end up being all talk and little action because my actual job is keeping my quite busy and I have some other hobby projects too, but let's see where it goes.

If you announce at the beginning of the discussion that you will not have time for the project, it does not look serious. You will need a lot of time in order to comeup with something serious.

Edited June 28, 2021 by AL|EN
typos

[Taylan]

@AL|EN maybe you could shortly summarize the fundamental flaws you have in mind? I don't use Discord unfortunately but I'd be very interested to hear your thoughts.

Regarding sound set installer only vs. the whole of PieDU, I'm currently planning to implement just enough of PieDU at first to allow creating mods of type "sound set" and nothing more, so I have something working and useful to be happy about, without immediately getting lost in complexities. At the same time I thought it would be best to start considering right away where else one might take the project, and start writing the code with that bigger vision, so there won't be many roadblocks in the future when it's extended to support more and more things.

As a counter-example, think about how WeiDU started as "dialogue utilities" and can now do everything. That organic growth from a tool focused on one task towards a tool that can do anything is probably the main reason why it became so messy, which is exactly what I'd like to avoid.

As for focusing on the players, modders, and ecosystem, I guess you people will have to help me on that since I'm new around here.(*)

For instance I didn't even know about Project Infinity until this very moment! That's certainly a very cool project and I wonder how I could shape my project for least duplication of effort and most compatibility between tools. I'll keep in mind @subtledoctor's suggestion and make sure that there's a mode of operation in PieDU where its interactive UI can be skipped (allowing you to call it with command-line arguments instead), so that Project Infinity could theoretically be used as a master GUI for WeiDU / PieDU / (insert other tool here).

One thing I believe I already understand about the players & modders, and you have to correct me if I'm wrong, is that the community contains both people who are very creative but don't have any programming knowledge, and people who are advanced/professional programmers. So that's something I'm keeping in mind. With the "mod types" I mentioned before, I intend to make it as easy as possible to create certain types of mods (ideally with zero code, just some definitions in a simple format), while also offering a rich Python API for programmatically editing any files you want.

(*) Since I said I'm new around here, here's a funny story about that actually: I think when I was somewhere around 14 years old, which means around 13 years ago, I very briefly toyed with a class kit mod for BG2, and that was one of my first interactions with something close to software development. I might have created an account here to ask a question or two, or maybe it was a different forum, I don't remember. Sadly I got bored with it soon and didn't touch BG modding again for all those years. Now thanks to the attention brought to BG by Beamdog and then the BG3 announcement, finally I am here again after becoming a proper software developer.

[subtledoctor]

1 hour ago, Taylan said:

I'm currently planning to implement just enough of PieDU at first to allow creating mods of type "sound set" and nothing more, so I have something working and useful to be happy about

Nothing wrong with that. Even if nobody ever uses Python for anything IE-related except soundsets, if this ends up being a narrowly-focused soundset utility, that's just fine. If it takes off and can be used for more general-purpose modding, that would be fine too, even if less likely.

1 hour ago, Taylan said:

I'll keep in mind @subtledoctor's suggestion and make sure that there's a mode of operation in PieDU where its interactive UI can be skipped (allowing you to call it with command-line arguments instead), so that Project Infinity could theoretically be used as a master GUI for WeiDU / PieDU / (insert other tool here).

I meant something more fundamental: maybe it would be a good idea to completely divorce the GUI stuff from the underlying modding library. Like, if someone ends up with a bunch of back-end functions that allow people to mod arbitrary aspects of the game with simple Python scriptlets, great! As mentioned, (Weidu has already been going down this road; a lot of mod written today are written entirely with higher-level function calls that are short and simple, whereas a lot of the old mods from ~15 years ago perforce used longer and more complex-reading low-level code. I've learned to write item alterations both with and without GET_OFFSET_ARRAY, and boy howdy is that thing handy!)

But maybe that should be the be-all and end-all of PieDU. Then, separately, someone could make "PieGU(I?)," which would be a GUI shell capable of installing both PieDU and WeiDU mods. Alternatively, AL|EN could enable PI to handle PieDU mods as well as WeiDU mods, and players could use PI if they prefer it to PieGU. (PI has a substantial lead, after all, and has neat bells and whistles like conflict detection and install order importation. It would take a long while to apply that stuff to yet another app. (Believe me.)

This kind of talk is pie in the sky (pun intended), but it's nice to zoom out and get a good glimpse of the broader perspective...

[AL|EN]

On 6/28/2021 at 8:32 PM, Taylan said:

@AL|EN maybe you could shortly summarize the fundamental flaws you have in mind? I don't use Discord unfortunately but I'd be very interested to hear your thoughts.

That's you saying "I want to build skyscraper but I won't read any engineering book that has red book cover"? Do you expect me to summarize 6 years of experiences, ideas and thoughts in a six lines? Here are the glimpse:

• auto-update system
• application model
• backup system
• install/uninstall routine
• debugging
• translation system

Let me stop here by saying that you have to start with good foundations, without 20 years of flawed design that is impossible to change because of compatibility with older mods. If you want to know more, the ball is in your court.

On 6/28/2021 at 8:32 PM, Taylan said:

As a counter-example, think about how WeiDU started as "dialogue utilities" and can now do everything. That organic growth from a tool focused on one task towards a tool that can do anything is probably the main reason why it became so messy, which is exactly what I'd like to avoid.

So you don't want to repeat WeiDU mistakes, good. Then you have to learn about all things that makes weidu messy so you don't repeat them. This is no established ground so any design flaw should be eradicated with fire at the begging. This will require strong opinions and making difficult choices. Also, if you inherit the part of the name from WeiDU, don't you think that you intrinsically drive your mindset into the wrong road?

On 6/28/2021 at 8:32 PM, Taylan said:

One thing I believe I already understand about the players & modders, and you have to correct me if I'm wrong, is that the community contains both people who are very creative but don't have any programming knowledge, and people who are advanced/professional programmers. So that's something I'm keeping in mind. With the "mod types" I mentioned before, I intend to make it as easy as possible to create certain types of mods (ideally with zero code, just some definitions in a simple format), while also offering a rich Python API for programmatically editing any files you want.

Regardless of how many 'tech savy' modders are out there, you should aim to create something that can be used not only for trivial task but also for advanced tasks and allow low-level modification only for extreme cases. All of this with limitations to not allow one line from previous mod to completely damage game resource so all other mods can't be installed.

@subtledoctor "About this, feelings correct you have."

That's pretty much what it's needed in order to not fall into weidu trap. Application model should have two things:

• a single system-wide GUI app that handles everything except actual game data modification like install order routine
• a multiple versioned system-wide app that handles game data modification

A nice foundation for unlimited growth. A man can dream...

Edited June 30, 2021 by AL|EN

[Magus]

I think installation/uninstallation are fine in weidu. There's really no much leeway in how to do it, given how IE works and the need to support a plethora of mods.

Translation is ok too. One way or another, it's going to be a bunch of files with numbered strings.

We already have an API to edit any files programmatically, it's iesh. The problem is making it compatilbe with weidu mods.

(And pretty sure there's no need to worry about GUI at the start.)

Edited June 29, 2021 by Magus

[Taylan]

@AL|EN I thought you were inviting me to discuss things over Discord. Is there some kind of document there that I can read? I just joined but just see chat. As a Discord noob I might be overlooking/misunderstanding something. I'd love to read some elaboration on what you mean with those points, e.g. how could/should an auto-update system look, what is meant with "application model" and so on.

About installation / uninstallation / backups: I wonder if it would be best to just take a "start from scratch every time" model, meaning: before any file is touched, a pristine backup of all files is stored in a safe place. Then, whenever the user wants to uninstall something, change the order of installation, etc., the process looks like: 1. restore the original untouched game, 2. reinstall everything in the new order.

What would such a method entail? As far as I understand, most game files (the BIF files in the data directory being like 75% of the game by size, I think) are never actually modified, and instead modified versions of individual files (like CHARSND.2da) are dropped into the override folder. However there are exceptions, like the TLK files which are modified in-place. Is there a guide somewhere explaining what files a mod *needs* to modify in-place vs. what can be done via the override folder? If the number of actually modified files is small, then creating a complete backup and starting from scratch every time should be easy and not waste too many disk space or computing time. (Although installing a behemoth like SCS takes ages even on my top of the line PC, so I guess I'd need to see if I can port it to PieDU and make it more efficient. I suspect that there's a ton of optimization possibilities / that WeiDU wastes way too many resources for the sake of simplicity. I dunno.) And then there's new files a mod drops in place, like sound files, which I guess could either be just left there to rot when not needed anymore, or would have to be registered by name and later cleared up...

And some questions about iesh, or more precisely the "infinity" Python module package it provides: how actively is it maintained, how battle-tested is it, and how suitable would it actually be for a modding framework? Are there any known design flaws that inhibit performance or force the programmer to write cumbersome code for things that should be easy? I'm sure there's a ton of good code in it, but I wonder if it wouldn't be better to write a library with modding in mind from the start, just copying bits and pieces from iesh where needed (or using its code as a reference point).

[subtledoctor]

1 hour ago, Taylan said:

However there are exceptions, like the TLK files which are modified in-place. Is there a guide somewhere explaining what files a mod *needs* to modify in-place vs. what can be done via the override folder?

Unless I'm forgetting something, I think dialog.tlk is the only file that is modified-in-place. Everything else is simply plopping a modified version of a file in /override. Of course the sticky bit (or anyway it was fairly sticky 15 years ago), is that sometimes a file in /override needs to be modified from its already-modified state - mods can't just grab it from /data every time, else the earlier mod's changes will be wiped out. So maybe the "plop in override" versus "modify-in-place" thing is a distinction without a difference.

1 hour ago, Taylan said:

(Although installing a behemoth like SCS takes ages even on my top of the line PC, so I guess I'd need to see if I can port it to PieDU and make it more efficient. I suspect that there's a ton of optimization possibilities / that WeiDU wastes way too many resources for the sake of simplicity. I dunno.)

You're never going to be able to convert old mods to your new system, so I wouldn't even worry about that. They will generally involve too many idiosyncrasies accounting for the way Weidu works. Very much especially SCS, I suspect.  (Unless SSL could be adapted to run from within PieDU?  But then, would it even make a difference given that the mod will still be using SSL?  I don't know.)  Whatever value you expect out of PieDU will probably have to be measured in new mods made with ti in the future.  How many will there be?  Is this idea even worth it?  Are we experiencing the last gasp of Infinity Engine modding?  Maybe.  (But then, we said stuff like that five years ago, and look what's been produced in the last five years...)

[DavidW]

I feel the need to defend the honor of WEIDU.

Here's how I'd code this task in WEIDU, assuming exactly the input format for descriptions.txt in Taylan's post:

 

Spoiler

DEFINE_ACTION_FUNCTION install_custom_soundset
    STR_VAR soundset_path=""
            soundset_definitions="definitions.txt"
            file_path="%soundset_path%" // default to same location unless overwritten
BEGIN

    // parse the definitions
    LAF parse_ini STR_VAR file="%soundset_path%/%soundset_definitions%" RET_ARRAY ini_array END
    
    // copy over the files, and get an array of tlk entries to add to the table
    ACTION_PHP_EACH ini_array AS k=>v BEGIN
        ACTION_IF FILE_EXISTS "%file_path%/%v%" BEGIN
            //install it
            COPY "%file_path%/%v%" "override"
            // get its non-suffix name
            OUTER_PATCH_SAVE resref "%v%" BEGIN 
                REPLACE_TEXTUALLY EXACT_MATCH ".wav" "" 
            END
            // get its associated text line, if any
            ACTION_IF VARIABLE_IS_SET $ini_array("%v%") BEGIN
                OUTER_SPRINT text_line $ini_array("%v%")
            END ELSE BEGIN
                OUTER_SPRINT text_line ""
            END
            // add to dialog.tlk
            OUTER_SET strref=RESOLVE_STR_REF ("%text_line%" [%resref%])
            // get the number for the command type
            OUTER_SET code=IDS_OF_SYMBOL (sndslot "%k%")
            OUTER_SET $soundset_write_array("%code%")=strref
        END
    END

    // add the entries
    
    LAF add_blank_column STR_VAR 2da="charsnd.2da" header="%ini_array_name%" entry="-1" END
    COPY_EXISTING "charsnd.2da" override
        COUNT_2DA_COLS colcount
        FOR (row=0;row<rowcount;++row) BEGIN
            READ_2DA_ENTRY row 0 colcount row_number
            PATCH_IF VARIABLE_IS_SET $soundset_write_array("%row_number%") BEGIN
                v=$soundset_write_array("%row_number%")
                SET_2DA_ENTRY row (colcount - 1) colcount v
            END
        END
    BUT_ONLY

END

It appeals to a couple of subsidiary functions which I haven't defined; they'd be about 5-10 lines long each, I think. It took about 20 minutes to code, and it's about the same length as Taylan's Python script (and without appealing to a hypothetical library of weidu functions). The syntax is a little more cumbersome than python, to be sure.

WEIDU isn't perfect, but it's not just nominally Turing-complete; it's a reasonably effective programming language, even if 95% of mods don't use it that way. Would it be nice to have object-orientation, anonymous functions, more sophisticated handling of arrays as values, or a less idiosyncratic syntax? Sure. But it's way more powerful and flexible than it gets credit for.

[DavidW]

(EDIT: actually, skip this one, it's overcomplicating things.)

[DavidW]

I'll stop after this, but here's a direct implementation of Taylan's design spec, in WEIDU rather than Python (again helping myself to a couple of auxiliary functions):

Spoiler

DEFINE_ACTION_FUNCTION add_soundset
    STR_VAR path=""
BEGIN

// # Initialize our theoretical WeiDU library
// # This would do some magic to find the game files and such
INCLUDE "%MOD_FOLDER%/lib/sfo_lite.tph"

// # Define mapping from human-readable sound IDs to game-specific integers
// no need: they're in sndslot.ids and WEIDU can read them directly

// # Read the definitions for the sound set
ACTION_FOR_EACH category IN config sounds subs BEGIN
    LAF parse_ini STR_VAR file="%path%/definitions.txt" category RET_ARRAY "%category%"=ini_array END
END

OUTER_SPRINT name $config("name")

// # Copy the WAV files to where they belong
ACTION_PHP_EACH sounds AS k=>file BEGIN
    COPY "%path%/%file%" override
END

// # In this dictionary we will save mappings from WAV file name to string reference
ACTION_CLEAR_ARRAY strRefs

// # Open the TLK file for our language
// no need, WEIDU opened it automatically

ACTION_PHP_EACH subs AS wav=>string BEGIN
  // # Add the string to the game, receiving its numeric ref
  // # Remember what WAV file name it belongs to
  OUTER_PATCH_SAVE resref "%wav%" BEGIN REPLACE_TEXTUALLY "\..*" "" END // strip the .wav
  OUTER_SET $strRefs("%wav%")=RESOLVE_STR_REF("%string%" ["%resref%"])
END

// # Save the modified TLK file
// no need to do this in WEIDU

// # Let's say the function parse2DA gives us a "2DA object"
COPY_EXISTING "charsnd.2da" override

// # We will build up a new column to add to the 2DA
     LPF add_blank_column STR_VAR header="%name%" entry="-1" END

// # The 2DA object lets us get the rows via the rows property
// # And each row in turn has a cols property
// WEIDU does this slightly differently
    
    COUNT_2DA_COLS colcount
    COUNT_2DA_ROWS colcount rowcount
    FOR (row=0;row<rowcount;++row) BEGIN
    
  // # The sound's game-specific integer id is in the first column
        READ_2DA_ENTRY row 0 colcount intId
  // # Get the human-readable sound ID for that integer
        LOOKUP_IDS_SYMBOL_OF_INT soundId sndslots intId
  // # Get the WAV file name for that sound ID
        SPRINT filename $sounds("%soundId%")
  // # Get the string ref for that WAV file name
        strRef = $strRefs("%filename%")
  // # Add that to the column we're building
        SET_2DA_ENTRY row (colcount - 1) colcount "%strRef%"
  
    END
  
// # The function .addColumn takes a column title and the list of values for it
// we don't need this in WEIDU: we've been working directly with the file

// # Install the modified 2DA file into the game directory
// That's happening automatically
    PRETTY_PRINT_2DA

END

 

[Taylan]

So as I've recently realized and is mentioned by @DavidW above, symbolic names for character sounds can be found in the SNDSLOT.IDS file. That's pretty neat.

However, are the symbolic names compatible between all games? I just compared BGEE, BG2EE, PSTEE, and IWDEE. They seem to be mostly identical, with exceptions. At the end of all but PSTEE there's some strange entries with the name of some game prefixed. Not necessarily the game the file is for, e.g. in BG2EE there's entries prefixed with BGEE_ and IWDEE_.

The IWDEE_ entries found in BG2EE seem to correspond to EE_ entries in IWDEE. I guess if a sound set really contains sounds for those ids, then at worst one would have to write some duplicate entries, e.g. the sound set definition would link both EE_LEADER2 and IWDEE_LEADER2 to the same WAV and all should be fine.

Insight from others welcome.

[Graion Dilach]

There are subtle differences between handling. Select_Rare5-8 doesn't exist in all games and IIRC Action7's slot differs.

https://github.com/Gibberlings3/EE_soundset_tool/blob/master/cd_soundsets/tph/cd_add_soundset.tph#L96-110 contains the order in the 2DA. Or just read the whole file because this is what you want to reimplement pretty much.

I haven't looked at PSTEE because that's on a hardfork AFAIK and this 2.6 update doesn't roll out to it.

Edited June 30, 2021 by Graion Dilach

[Taylan]

A sound not existing in all games or having a different numeric ID is no problem.

1. If a sound set provides a sound, but it doesn't exist in all games, it will simply not be installed for that game.

2. The numeric ID will always be looked up from the current game's SNDSLOT.IDS, the mod utility never assuming any knowledge of numeric IDs.

What would pose a very minor problem:

- Games use different symbolic names for what's obviously the same sound. E.g. if one game calls it BATTLE_CRY_1 and another calls it BATTLECRY1. Easy solution: sound set definition must define both, pointing to the same WAV.

What would pose a significant annoyance:

- The same symbolic name is used for completely different purposes in two games. E.g. if the name "DAMAGE" is used by one game for the generic sound whenever a character is hit, but another game uses it for the "critical damage reached" sound where the character asks for help. (This particular example is surely not the case, it's just an example of what could be.) Solutions might be: 1. use a symbolic name that exists in none of the games, the installer translates it to the correct real name; 2. sound set definition must specify which game is meant with the symbolic name, by specifying it as e.g. "DAMAGE/BG2EE" and "DAMAGE/IWDEE" and the installer interprets it the right way.

That last issue *might* be the case for some symbolic names that are used for NPCs only, but I think it never applies to a regular character sound, so I think we're safe. This means the installer doesn't have to include tables mapping symbolic names to integers; it can use the mapping from SNDSLOT.IDS and expect the sound set definition to use names from there.

The only "special knowledge" a sound set installer needs is a mapping from sound to WAV filename suffix, for pre-2.6 compatibility, because that mapping isn't defined in any game file, correct me if I'm wrong?

[kjeron]

 

1 hour ago, Taylan said:

2. The numeric ID will always be looked up from the current game's SNDSLOT.IDS, the mod utility never assuming any knowledge of numeric IDs.

...

The only "special knowledge" a sound set installer needs is a mapping from sound to WAV filename suffix, for pre-2.6 compatibility, because that mapping isn't defined in any game file, correct me if I'm wrong?

It's not defined in any game file, for any EE game, unless you're talking about the EXE.

Index 75 will always point to the 75th sound index (offset 0x1d0 of the CRE file), no matter what label it has in SNDSLOT.IDS.  Which event triggers it to play is determined only by the EXE (and sound options), and yes some of the same indexes are used for different purposes in different games.

That IDS file is for assigning arbitrary labels to hardcoded indexes.  Renaming the label won't change the index's function or existence, though it may change which index a uncompiled script (dlg/d/baf file) points to when it's compiled/processed (assuming it was using those arbitrary labels instead of indexes in the first place).

In contrast to IDS files such as INSTANT.IDS, for which an entry's mere presence has function, you can overwrite SNDSLOT.IDS with a blank file and sound sets will continue to function as if nothing changed.

You're better off ignoring what's in SNDSLOT.IDS, and mapping it yourself with custom labels (per game) for each sound.  Use CHARSND.2da to determine if a slot is used by the game, not SNDSLOTS.IDS.