Things that prevent WeiDU successor or alternative?

[AL|EN]

Hey,

This is a long overdue post, let's have a loose discussion about this. What do you think are the core bricks that prevent WeiDU's successor or alternative?

My own would be:

• It's too late for a new thing to attract attention, no matter how good it would be.
• It's too much work to learn new, shiny things even just for fun.
• WeiDU interoperability with dialogs added from other mods, you either recreate it or you can forget about adding cross-mod content.
• WeiDU backup system, you either recreate it completely or fail to uninstall/reinstall a mod.
• Lack of accompanying tools that are specifically created for WeiDU.
• Lack of a mod manager that can install WeiDU mods along with mods that are using other 'modding frameworks'.

Those are some loose examples from my memory. I'm not sure about the weight of each thing, or how it affects the subject and I would love to hear your thoughts!

EDIT: I've added 'or alternative' to not limit the potential scope of the discussion.

Edited March 12 by AL|EN

[mickabouille]

1 hour ago, AL|EN said:

WeiDU backup system, you either recreate it completely or fail to uninstall/reinstall a mod.

For me that’s the only real blocker.

In my (really limited more proof-of-concepty) experiments with this, I just generate a weidu mod that I then install to keep weidu happy.

Never went really far, really, just generating fixup minimums without needing to remember the baroque syntax of tpX files.

[subtledoctor]

Honestly, I question the premise. I don’t know that Weidu really needs a “successor.” 

Weidu started very simply: allow multiple mods to patch dialog.tlk sequentially. Tack on some basic file-editing operations and you have a mod installer. 

Any replacement would still need to have sequential .tlk changes and sequential file patches/overwrites, and would need to parse existing .tp2 scripts. So the replacement, under the hood, would need to look a lot like Weidu. And I don’t see a ton of daylight between a nice GUI that looks like Weidu under the hood, and a nice GUI mod manager front-end that actually uses Weidu under the hood. Like PI or Argent77’s Weidu Installer. These tools already exist and have a head start; seems more productive to put effort into perfecting them, rather than creating a ground-up replacement. 

[paladin84]

As a novic modder (while having some experience with other programming languages) I can tell, that weidu is horrible. You have to study the whole new languages that doesn't look like any of the mainstream programming languages - and this is tim econsuming. I still striggle with simple things. Something python based would be so much better.

But unfortunately I have to agree with all the arguments to keep using weidu. It doesn't make any sense to create a new tool now.

[Incrementis]

I'm also a newbie modder and agree with the part paladin84 wrote about WeiDU's usability. I would also like to add that it bothers me that WeiDU is unable to manipulate hard-coded things (I know EEex can do this, but I think a successor should have it too. It will make things more complicated and therefore more complex to use more than one tool and language)

3 hours ago, AL|EN said:

WeiDU backup system, you either recreate it completely or fail to uninstall/reinstall a mod.

This is something I need to test the mods. If the successor doesn't have that, I don't know if I want to use it.

All in all, I will continue to modify and share when possible, but very slowly. It's up to you knowledgeable and experienced modders which tools I will use until I also reach that skill level to create and contribute to tools(this will take years, maybe a decade?).

[jmerry]

The games just weren't designed to be easy to mod in the first place. Everything important is in binary files with bespoke specifications, and that means you need the specialized tools to get at them. By way of contrast, I've done some stuff with the Civilization games too - and there, you're using established formats like XML and SQL to edit databases, or scripting in something like Python or Lua to get at events that the database-editing can't reach. Plus the environment of people installing multiple mods that interact with each other is just really challenging; it's amazing things work as well as they do with the system we have for the IE games.

So as long as we keep playing these games in their creaky old engine, WeiDU is how mods will be done. Is there room for improvement in how the players see this - a better interface for installing the mods themselves? Sure. But you're not going to replace the back-end. Modders just have to suck it up and learn how to work with an unusual programming language.

[Daulmakan]

50 minutes ago, jmerry said:

Plus the environment of people installing multiple mods that interact with each other is just really challenging; it's amazing things work as well as they do with the system we have for the IE games.

I wanted to highlight this. Weidu has spoiled me when trying to install multiple mods in other games. While several others have done something similar with a monolithic install tool (unity mod manager, TSL patcher, etc), no other game I play has both the sheer amount of mods together and the stability that weidu/the IE community have provided.

[CamDawg]

Honestly, we're about 15 years too late for a switchover. Most old mods are lucky to get maintenance (including WeiDU itself); realistically there's no way they'd be migrated to a new tool. So we're already starting from a point limiting any new tool: it would have to be backwards compatible,  which is already a lot of extra work and design constraints.

[suy]

I think this is something that pops up from time to time, and everyone has their own priorities or preferences. The last time I've read about a proposal, some of the answers were appalling, so no wonder that whoever wants to do this ends up burned.

I have done enough weidu so far myself, that I know that I want to reduce it as much as possible for my own use cases, and I've started to do so. My plan is not to make an alternative from the start, but to work on phases.

1. The first phase is to create a tool that can be invoked manually, ahead of mod install time, to create or alter game files. The files can be added to the mod, as "static" files, like if you created them in NearInfinity, but instead of pushing buttons, typing code. This exists in a few forms. I think iesh can do this, and dark0dave's library as well. There are few libraries/tools published online that have done similar things in other languages.
2. The second is to call that tool from weidu, when the mod installs. This is also something that exists. Stuff like tisunpack, iconv, etc. Stratagems Scripting Language (SSL) uses Perl, if I remember correctly. This is just a helper to do heavy lifting of things in which weidu is not so well suited. It needs to delegate to weidu for copying the files, so the files end in the backup, if I'm not mistaken.
3. Next would be the same as the previous, but more intensively. The whole mod component would be coded in the external tool. Then, to install this mod component using the new stack, it would not need weidu (but you won't get uninstallation support, unless this new system is last in the installation). This could be enough for a quick and dirty tweak at the end.
4. Same as before, but the external tool could impersonate as weidu, for compatibility. I'm not sure how much would be useful or necessary, but I'm thinking at least writing in the weidu.log, and writing the backup files in the same format, so weidu could uninstall the "foreign" mod without noticing it's not written in weidu. I'm guessing this might be pretty hard, as the foreign tool it might need to generate a tp2 for compatibility, or maybe weidu should receive some feature to detect what to do.

Now, this is a lot, and I don't see it happening realistically (and few more steps would have to come after, but that's even beyond what I'm gonna bother writing).

However, despite it, I've started to work on something, and I'm only at stage 1, but I'll soon reach the 2nd, and I expect to reach close to the 3rd at some point. It's unlikely that I'll go myself beyond that (which, as I said, I think it's not too different than the status quo, just in a different angle). But my use case is very niche. I only have immediate plans to support ITM, SPL, EFF (most of it it's done), and later on maybe CRE and STO. And I'm only worrying about a mod that comes probably the last in the install order to ease development, so that's why I'm in a position of attempting to do any of that madness. Not being able to be compatible with the existing ecosystem is super painful, and not a realistic proposition to the community, but if it's self inflicted pain, could be Ok.

I don't know how much this answers the original question, but to summarize, I think the main deal is the general compatibility with (un)installation procedures. I've only glanced at what weidu does for backups, and it seemed reasonable to replicate in another software. But the whole workflow of weidu reinstalling B and C when A gets updated since too much work to fix, that I don't expect this happening, unless someone wants to contribute to weidu to make it aware of other tool.

[CrevsDaak]

I don't think it's hard to code in WeiDU, even if it has its oddities and all, the language is pretty powerful too and there's a lot of existing code that can be read in order to learn. Also any language that would allow for modification of the game's files would evolve into a similar level of complexity, or else it would fail to replicate existing capabilities which would be unacceptable. Very few things warrant for external, non-WeiDU programs to be used (tileset conversions, audio decoding, creation of huge scripts), and even less mods require the biggest benefit of external programs which is how much faster they do things.

WeiDU is also specifically designed to deal with binary files which many other systems do not do as well, WeiDU handles binary files better than every single other language I've ever attempted to put up to the task. The actual headache that it is to deal with such things from C and such, WeiDU is the pinnacle of imperative programming aimed at binary formats: it does exactly what you want it to do, exactly at the offset you want it to. No need to deal with buffers, endianess, seeking on a file, mismatched integer types, etc. Sure you can write a library in C, but that doesn't detach you from the compatibility problems inherent to it, nor different platforms make WRITE_BYTE behave any different.

As for modifying binary files using text commands rather than editing the binary data as such, there's DavidW's SFO 2e which while it's rather hard to guess how to do some things with, offers insanely valuable functionality on most of the game's file formats and lets you perform very complex changes with relative ease.

5 hours ago, suy said:

The first phase is to create a tool that can be invoked manually, ahead of mod install time, to create or alter game files. The files can be added to the mod, as "static" files, like if you created them in NearInfinity, but instead of pushing buttons, typing code. This exists in a few forms. I think iesh can do this, and dark0dave's library as well. There are few libraries/tools published online that have done similar things in other languages.

You can also just do this on WeiDU and create code that won't be present in the final version of your mod. I despise interacting with binary files and have been doing this for my mods for quite some time. Being able to use WeiDU's existing codebase and functions is a huge advantage as compared to having to reinvent the wheel to modify an item or two (or the existing alternative of clicking buttons around for several hours). Having code to create the binary files and then some other code to handle install-time tasks seems like the best idea to me, I should probably make it package the entire mod on its own as well, so it just creates new releases without me ever having to touch the release version files.

[suy]

1 hour ago, CrevsDaak said:

Being able to use WeiDU's existing codebase and functions is a huge advantage as compared to having to reinvent the wheel to modify an item or two (or the existing alternative of clicking buttons around for several hours).

Very hard disagree, and I'll give you an example. The one that made me decide that I've wasted too much time with weidu, and that I'd attempt to use something else.

Try to make a copy of SPCL822 (the spell that makes a Monk get a base AC going lower with levels), and patch it with ALTER_EFFECT/ALTER_SPELL_EFFECT so that every effect sets AC to a value lower than what it's there by default. You'll hit tons of limitations due to the values being negative for parameter1. I for one, could not know how to do it, and I only found a weidu bugreport, and an alternative version of the function that DavidW posted. Since David's version has the limitation of only allowing till -9, and I needed to go even lower, I gave up, and altered the item with NearInfinity, then added the generated file to the mod. CamDawg published another update/alternative to the function, but it was after I had to solve it "manually". That was posted 15 months ago, December 2022. I don't get the impression that the weidu ecosystem is so well established. In my experience, and for the mods that I want to make, weidu is not satisfactory, at all, and I keep seeing the wheel being reinvented constantly (as the links show). Maybe it's awesome for dialogs and scripts, I don't know, but I am very skeptical.

[CrevsDaak]

11 minutes ago, suy said:

I don't get the impression that the weidu ecosystem is so well established.

It definitely does not cover every case, but you can easily write code to cover the odd cases (particularly on something as trivial as patching effects on a spell that's as neat and well formed as this one). Because there's C libraries with bugs it doesn't mean every C library out there is bugged, and the same applies to limited functionality on WeiDU functions, since the language does lack a null value and suffers from that.

COPY_EXISTING spcl822.spl override
  GET_OFFSET_ARRAY ab_array SPL_V10_HEADERS
  PHP_EACH ab_array AS int => ab_off BEGIN
    GET_OFFSET_ARRAY2 fx_array ab_off SPL_V10_HEAD_EFFECTS
    PHP_EACH fx_array AS int => fx_off BEGIN
      PATCH_IF (LONG_AT fx_off) == 0 BEGIN
        READ_LONG (fx_off + 0x8) param1
        WRITE_LONG (fx_off + 0x8) (param1 - value)
      END
    END
  END
BUT_ONLY

Doesn't take long at all to write, doing this from another language will always be more convoluted than learning how to do it in it. You could even check the levels on each header and apply the modification to the effect non-linearly.

[suy]

1 hour ago, CrevsDaak said:

Doesn't take long at all to write, doing this from another language will always be more convoluted than learning how to do it in it.

In another language it would be very similar to the following pseudocode:

spell = Spell::read(file)
for effect in spell.effects
	effect.parameter1 -= 1
end
spell.write(file)

 

[CrevsDaak]

Does that language already exist though? You rely on a pre-defined library of the IE formats which is something that's not fun at all to write. Meanwhile, editing spells with WeiDU doesn't require me to write any code except what actually impacts the spells themselves. That's why I called a new language more convoluted, the amount of work it requires to match current WeiDU capabilities, rather than it not being easier to use  (as it WILL be easier to use by means of further abstraction). I might sound too critical of the idea but I would have no reason not to use such a tool either, and if I thought it necessary I'd work on something like it, but dealing with WeiDU directly is just more efficient.

As far as the more complex IE binary formats go, external tools would be much more useful than WeiDU, which is why I don't see it a worthwhile effort to make modification of spells and other simple formats easier when WED files or other aspects of area creation are still as much of a nightmare to deal with.

[subtledoctor]

3 hours ago, suy said:

Try to make a copy of SPCL822 (the spell that makes a Monk get a base AC going lower with levels), and patch it with ALTER_EFFECT/ALTER_SPELL_EFFECT so that every effect sets AC to a value lower than what it's there by default.

This is a problem with a secondary function, not with the language itself. GET_OFFSET_ARRAY was designed for just this kind of example. It seems you are trying to use a hammer to drive in a screw. 

3 hours ago, suy said:

Maybe it's awesome for dialogs and scripts, I don't know, but I am very skeptical.

Considering you admittedly don’t have experience with this, and the language is literally called Weimer’s Dialogue Utility (IIRC) I’m not sure what the source of your skepticism is. The file formats of IE assets are well-mapped; I feel like you could use any language to do basic file patching. But what language is equipped to create dialogue interjections in a quest or NPC mod? Or modify cut-scene scripts?