Things that prevent WeiDU successor or alternative?

[jastey]

2 hours ago, subtledoctor said:

you could use any language to do basic file patching. But what language is equipped to create dialogue interjections in a quest or NPC mod? Or modify cut-scene scripts?

This. We don't even have to think about a successor if it's not capable of patching dlgs and scripts like weidu does.

Maybe I just don't get the point of this discussion.

[suy]

6 hours ago, CrevsDaak said:

Does that language already exist though? You rely on a pre-defined library of the IE formats which is something that's not fun at all to write.

I already said so in the previous comment, yes, I've seen a handful of libraries in a few different languages, none of them having gained any traction. And I have started one more that it's already working to some degree, and I will make it public in hopefully the next months (and I also said I don't expect to gain any traction, nor to be a replacement, just a complement of the current ecosystem). It's not fun to write such library for me, no, as the file formats are many and have lots of fields. But I'm having even less fun trying to do reasonable things in weidu, and not being able to. If you are happy with the status quo, better to you. I don't plan to subtract anything from it (nor I could, even if I wanted).

5 hours ago, subtledoctor said:

This is a problem with a secondary function, not with the language itself. GET_OFFSET_ARRAY was designed for just this kind of example. It seems you are trying to use a hammer to drive in a screw.

I very much disagree. There would be no need to write such a function if weidu supported data structures. In fact, given that weidu is only capable of doing reads and writes via offsets, it's weidu's hammer which is constantly on the look for nails. I refer to my example again. The pseudocode snippet is based on real code. If you prefer the code using GET_OFFSET_ARRAY, more power to you. You can rely on tools that already exist. I don't, and that's I have to suffer making my own library, and be able to read and write files the way I think makes sense and which is scalable and maintainable. Becaue I very much would like to write this:

for effect in item.effects
    if effect.code == 216
        print("I drain", effect.parameter1, "levels")
    end
end

Rather than:

  GET_OFFSET_ARRAY ab_array 0x64 4 0x68 2 0 0 0x38
  PHP_EACH ab_array AS int => ab_off BEGIN
    GET_OFFSET_ARRAY2 fx_array ab_off ITM_V10_HEAD_EFFECTS
    PHP_EACH fx_array AS int => fx_off BEGIN
      READ_SHORT fx_off fx_type
      PATCH_IF fx_type = 216 BEGIN
        READ_LONG fx_off + 0x4 amount
        PATCH_PRINT "I drain %amount% level(s)"
      END
    END
  END

But if you prefer the latter, that's great. The latter exists and it works, available for every modder. The former needs work still, but I plan to publish it this year. And it's not coming to threaten anyone. It's just like SSL. Something written in another language, called from inside weidu.

 

5 hours ago, subtledoctor said:

Considering you admittedly don’t have experience with this, and the language is literally called Weimer’s Dialogue Utility (IIRC) I’m not sure what the source of your skepticism is. The file formats of IE assets are well-mapped; I feel like you could use any language to do basic file patching. But what language is equipped to create dialogue interjections in a quest or NPC mod? Or modify cut-scene scripts?

I know very well of weidu's origins, and I know what weidu's name means (everyone who moves the mouse over the word in this forum gets a tooltip saying so! ). Indeed, I mentioned dialogues for this very reason. I have not much interest in dialogue editing, and I don't plan to work on it. I'm not claiming that any language is or will be equipped with tools to patch dialogues out of the box. Quite the contrary. I'm just explaining that, for the kind of mods that I want to write (which involve ITM, SPL, 2DA... but not dialogues), weidu has shown to be very, very frustrating. I'm explaining what I'm doing to overcome that frustration. I'm trying to do so in the context of AL|EN's question, which is what would be needed to have a weidu alternative.

As I explained in my first comment, I think if one would want to fully replace weidu (which, let me insist, I'm not saying it's likely or possible in practical terms), said person would have to start with something small (using it less and less for mods just dealing with a handful file formats, like the ones I do), then eventually grow, if people like the approach, there is time, etc. etc. etc. For a tweak mod, this is manageable, because it could come last (no backup? no big deal to do it manually), and the file types to support are few (and are the ones that weidu doesn't support well IMHO, but you might disagree on this).

And again, I'm not saying that any well established mod should be be rewritten to drop weidu "just because". I'm not saying it's doable or advisable, and I'm definitely not saying that I volunteer to do any more work than what I mentioned. And I definitely don't want to lose all the wonderful mods that I don't want to code, but that I use for playing. That said, in the unlikely case that a project like mine gathers any interest (and others do exist! it doesn't have to be mine!), the fact that it originally comes with no support for dialogs, scripts, etc, does not mean that it can't be added, or that it would be hard to do so. It's way, way, way harder that weidu implements data structures or supports better functions, etc.

PS: yesterday I wrote that first comment at 1 AM. Both yesterday and today I feel tired, so it's perfectly fine that I did not do my best at communicating what's my intent (to be fair, it's a difficult, nuanced topic that also involves a lot personal preference and social dynamics). I love this community and I'm just not happy of the technical limits of some tools. That's it. That's the gist of it. Cheers.

Edited March 13 by suy

[paladin84]

I would say dialog and script files are ok and I am not sure that much of the improvement can de done here. But binary files patching, copying, etc definetely can be improved (legacy argument aside).

[subtledoctor]

I hear you @suy. And I’m not sure we really disagree very much. I mostly do tweak modding and I have very much felt the sting of inadequate ALTER_/CLONE_EFFECT functions, and I try to stay away from GET_OFFSET_ARRAY as much as possible because it is annoying. So I do not mean to defend Weidu as a designed solution for this work. The biggest difference between you and me is that I have no programming background, so it doesn’t occur to me to think “X is better than Weidu.” And I fully understand that informs my bias. 

But I (naively, perhaps) don’t think that means we are “stuck with” Weidu. Seems to me, if there were Infinity Engine libraries in… Python or Ruby or whatever the kids use these days - maybe a modder could perform file manipulations in Python and drop the files in a temp directory. Then wrap that in a Weidu installation script that copies the files from /temp to /override. Now newer mods could be made with better/easier tools, old mods would still work, Weidu itself could be used for DLG patching, and install logs and backups/uninstallation would still work. 

Edited March 13 by subtledoctor

[CrevsDaak]

1 hour ago, suy said:

But I'm having even less fun trying to do reasonable things in weidu, and not being able to.

In part the lack of dedicated tutorials to niche aspects of it and documentation aimed at doing complex things, while also explaining them in a detailed manner, is a bit of an issue, I've already thought of pairing it with rather unknown things the game engine has to offer and comitting myself to writing explanations of sorts about it.

Your arguments are reasonable, and consider that I don't oppose to anybody writing IE libraries to deal with the game's files, if anything I'm more interested in what could one day be done with something like that, the issue is very much writing something like that myself, as WeiDU works well for what my needs are (not without making me lose some hair every now and then, but it's been a kinder experience after moving away from ADD_KIT).

And as far as running utility tools during installation goes, it can go very far, a very powerful aspect is being able to call it with the file data itself during the patch operation, and then deleting the whole buffer and writing the whole new file. This lets you basically do it natively in WeiDU except you modify the file with an external tool.

[DavidW]

18 hours ago, suy said:

In another language it would be very similar to the following pseudocode:

spell = Spell::read(file)
for effect in spell.effects
	effect.parameter1 -= 1
end
spell.write(file)

 

In SFO it looks like this:

spl.edit[SPCL822]
[
	m.ab_fx.alter{s_parameter1 = s_parameter1 - 1}
]

 

[DavidW]

Some belated thoughts:

1) The obvious constraint in a WEIDU replacement (for the moment setting aside the question of whether we want one) is that the existing mod ecosystem is coded in WEIDU (and, at this stage in the community's life cycle, the existing mod ecosystem is much bigger than the mods that will be created in the future). That means any WEIDU replacement will have to be able to handle an install order containing WEIDU mods, and hence must have a complete implementation of WEIDU inside it. That's a horrendously large overhead on what is already a horrendously large task. (Doubly so assuming the replacement is not to be coded in OCAML, which I assume most people wouldn't want, and so has to reimplement WEIDU and not just borrow code.)

2) What is *possibly* realistic is to have WEIDU call out to some external tool (using AT_NOW, I guess) to do file edits, rather than doing them natively. I think that's what Suy has in mind. And certainly there is lots of high-level functionality that WEIDU can't do, or can do only clunkily, using its base tools and the core function library it ships with. The question I'd ask is: what is the advantage of implementing that high-level functionality in a new language that gets called from WEIDU, rather than as functions within WEIDU itself. After all, WEIDU is a fully functional programming language. It has its disadvantages (its syntax is sometimes clunky; its variable typing is rudimentary; it doesn't allow functions as first-order values; it lacks a large library of preexisting resources) but its low-level functionality is very tightly connected to IE mods and of course using it natively avoids a lot of awkward interface issues between WEIDU and an external tool.

3) Ultimately I think the problem with a new tool that replaces WEIDU's functionality is the same as the problem with a rich function library within WEIDU: building and maintaining that resource is a huge outlay of time and no-one has been willing to put it in. My 'SFO', especially in its newer form, is the closest I'm aware of, and I think solves most of the problems that WEIDU critics raise, and one of these days I may get around to finishing documenting it and making it available to others to use, but ultimately I don't have that much free time for modding and what I have is mostly devoted to my own mods rather than maintaining community tools. I think that problem generalizes beyond me.

4) Part of people's problem with WEIDU is just unfamiliarity. This isn't a good reason to look for change. Programming concepts are largely transferrable between languages and the time taken to learn a new language is short compared to the time taken to implement a complicated project. It is true that the resources to learn WEIDU are limited (my Course in WEIDU was an attempt to address this) and that reading mods is a very imperfect way to learn it (90% of WEIDU mods use only 10% of WEIDU's capacities).

[DavidW]

11 hours ago, suy said:

Becaue I very much would like to write this:

for effect in item.effects
    if effect.code == 216
        print("I drain", effect.parameter1, "levels")
    end
end

Rather than:

  GET_OFFSET_ARRAY ab_array 0x64 4 0x68 2 0 0 0x38
  PHP_EACH ab_array AS int => ab_off BEGIN
    GET_OFFSET_ARRAY2 fx_array ab_off ITM_V10_HEAD_EFFECTS
    PHP_EACH fx_array AS int => fx_off BEGIN
      READ_SHORT fx_off fx_type
      PATCH_IF fx_type = 216 BEGIN
        READ_LONG fx_off + 0x4 amount
        PATCH_PRINT "I drain %amount% level(s)"
      END
    END
  END

 

I should do this one too, I guess:

[
	m.ab_fx.alter{PATCH_IF s_opcode=216 BEGIN PATCH_PRINT ~I drain %s_parameter1% levels~ END}
]

 

[RoyalProtector]

I have sometimes wondered if I could use WeiDu to call some Java or C++ code I wrote to do some task that I can do with those languages much more easily, while retaining the advantages of installing a WeiDu mod. Oh well.

At least it would be nice if there was a WeiDu "library" that wraps convoluted low-level code for common tasks (or maybe not-so-common ones).

(Disclaimer: I know very little about WeiDu -- I only dabbled briefly, so pardon me if I'm unaware of something modding veterans would know)

Edited March 14 by RoyalProtector

[argent77]

1 hour ago, RoyalProtector said:

I have sometimes wondered if I could use WeiDu to call some Java or C++ code I wrote to do some task that I can do with those languages much more easily, while retaining the advantages of installing a WeiDu mod. Oh well.

That is certainly possible and already actively used for certain tasks, e.g. for converting tilesets. More general operations, like patching game resources, requires more effort though to keep it consistent with WeiDU's backup and caching mechanisms.

[suy]

16 hours ago, subtledoctor said:

But I (naively, perhaps) don’t think that means we are “stuck with” Weidu. Seems to me, if there were Infinity Engine libraries in… Python or Ruby or whatever the kids use these days - maybe a modder could perform file manipulations in Python and drop the files in a temp directory. Then wrap that in a Weidu installation script that copies the files from /temp to /override. Now newer mods could be made with better/easier tools, old mods would still work, Weidu itself could be used for DLG patching, and install logs and backups/uninstallation would still work. 

Exactly! That's the 2nd step that I mentioned in my first comment of the thread. It's something done already in some components of existing mod (using AT_NOW to call something external, but it would like tisunpack, iconv, perl for SSL, etc.), and I know that my mod doing for ITM or SPL files it is gonna be seen eccentric (because those other mods do it for much different purposes), but I want to try it, and see how it goes. Maybe the approach has to be thrown away, but I need to attempt it first.

15 hours ago, CrevsDaak said:

In part the lack of dedicated tutorials to niche aspects of it and documentation aimed at doing complex things, while also explaining them in a detailed manner, is a bit of an issue, I've already thought of pairing it with rather unknown things the game engine has to offer and comitting myself to writing explanations of sorts about it.

[...]

And as far as running utility tools during installation goes, it can go very far, a very powerful aspect is being able to call it with the file data itself during the patch operation, and then deleting the whole buffer and writing the whole new file. This lets you basically do it natively in WeiDU except you modify the file with an external tool.

Indeed, the documentation that comes with weidu is a bit lacking some times. I of course read it, and give it a try, but it's the stuff posted in this forums that has saved me, because it is clearer what the intent is. Modding for having fun playing the game is great, but I have little patience this days, so if developing the mod is too frustrating, it feels like what I do for work, and give up easily. Stuff like CamDawg's post on item patching got me started with my first 55 lines of weidu, and after DavidW's course, it grew by 500 lines (literally, I see the growth just days after it was published, according to the git history).

If you have the time to write anything documentation related I definitely would appreciate it, a lot. The stuff about GET_OFFSET_ARRAY that you posted was helpful already, and if you have any hint for doing what you mentioned in the last paragraph, it's also welcome. It's possible that running an external process for each file in a loop is prohibitively expensive, though, but again, I'll have to try and see it.

[lynx]

50 minutes ago, suy said:

It's possible that running an external process for each file in a loop is prohibitively expensive

That's unfounded, unless of course the external process itself is inefficient. If you install 10pp, a full-blown perl binary will be executed on potentially hundreds of files individually. WeiDU/OCaml just doesn't have the required capabilities to do what needs to be done natively and more time is spent on the extraction, decompiling and compiling anyway. But there's also no barrier to doing the looping inside your platform of choice instead.

[suy]

Thanks for your replies and comments, @DavidW. Some small comments:

1. The examples of SFOv2 are more or less what I expected that could look like in terms of brevity and readability. I've seen SFOv2 before when I dug into the small issues that I've found in ToF, and which I reported on the forum. In some I even reported what my guess was of what the fix (or part of the fix) looked like. But as you said, time is scarce for everyone, so writing documentation for it has less priority than other stuff, so it's difficult to say if it will get some adoption. Until I started to search for weidu files in a ToF installation, I did not realize that such "special" syntax seems to be compiled into a file in weidu_external, which actually looks like the weidu code I'm used to. Then, the special syntax gets passed to functions, but still in string form. That's a bit worrying for someone who is not used to any of this, in that, it seems hard to realize how things really work inside the library, and figure out what's wrong when something doesn't work in the mod using the library.
2. WRT the second point, yes, calling something via AT_NOW is what I have in mind. In that point you said: "The question I'd ask is: what is the advantage of implementing that high-level functionality in a new language that gets called from WEIDU, [...]". My answer is that I don't know if the alternative approach is a net benefit or not, but until it is attempted and it gets tested, we don't know for sure. For example, I surely can be more productive doing that in the alternative language, given the circumstances. Is it worth the cost of having to write support for a few file formats in a general purpose programming language, and the possible slowdown? Depends. The implementation of the file formats can be used for other purposes. For example, I use extensively a program that comes with GemRB's iesh called ieparse.py, to inspect files in the console. I can verify if patching code done with weidu does exactly what I wanted, because I can convert the before and after to text, and diff them. For the binary files that I (or anyone else) added to git, I see them as text instead in the git log, because that tool is automatically used as a filter. I already have to implement some of those file formats in C++ for Moebius Toolkit, as in the future I want it to be an alternative to EEKeeper, which is abandoned. Sadly I don't see a way to reuse that C++ implementation for anything else, but another language might have a chance.
3. WRT the size of the existing mod ecosystem, and fully replacing weidu: I sincerely hope that we can keep running weidu for long. I think it won't be an issue on Linux, but I heard some concerning comments on weidu being still based on an old OCaml version, that might not be supported in newer Windows versions. I don't even remember who said it or where, so I can't double check if I'm misremembering, but it is very concerning, because some improvements here or there in weidu would be nice, but keeping it running is even more important. If maintaining or contributing to it gets even harder, then it can be a huge problem.
4. I think that fully replacing weidu is indeed, just too much to ask. But (related to the previous point), if it can be ehnanced a little bit to allow for more compatibility with a theoretical 2nd tool (I don't know how, and I don't see much need, but mentioning it for the sake of the discussion, as I think it was AL|EN's point of the post), could be enough. I don't mind much having to write a TP2, and make a mod that it's weidu on the shell, but maybe more of another software on the core. But I don't know about modding other games, so I suppose AL|EN was having in mind some other additional features to make a GUI or any other kind of installer.

Thanks!

[suy]

15 minutes ago, lynx said:

That's unfounded, unless of course the external process itself is inefficient. If you install 10pp, a full-blown perl binary will be executed on potentially hundreds of files individually. WeiDU/OCaml just doesn't have the required capabilities to do what needs to be done natively and more time is spent on the extraction, decompiling and compiling anyway. But there's also no barrier to doing the looping inside your platform of choice instead.

Oh, great to know then! You mean that the looping over the files calls one Perl instance on every loop? Is that working fine for Windows as well? I'm asking because my understanding is that forking and calling external processes is extremely efficient on most Unix systems, given how prevalent shell scripting is (and how many process those spawn in complex pipelines), and also that POSIX sort of implies, but that Windows just favors other kind of techniques.

I'll look at 10pp, thank you! I wasn't aware of that implementation detail.

Edited March 14 by suy

[lynx]

Of course it works, most users are still on Windows. Perl is invoked just to do the actual regex magic and then immediately give back control to WeIDU. It's the reason AT_NOW has a return value in recent versions.