Things that prevent WeiDU successor or alternative?

[CrevsDaak]

10 hours ago, AL|EN said:

Correct me if I'm wrong, you specifically have 'an installation routine as of:

You can greatly reduce your input to achieve this by doing something like

weinstall mymod --no-exit-pause --force-install-list X Y Z

and also possible to call the game from the same command as well (or just press F5 on NI and then waste time navigating to what file you want to look at). So it hardly takes much effort at all once you have the right setup. I normally print the relevant parts of the files I'm modifying from WeiDU itself into the console and just look at that. For more complex stuff like effect ordering and such nonsense you're pretty much forced to look at it from NI and test it in-game (particularly for effect reapplication order and some other utter nonsense), but it's not worth doing after every little change. Once you're used to how this works it's not troublesome at all, and takes considerably less than for clang to compile my shit.

3 hours ago, AL|EN said:

It's about the fact that you can't use it elsewhere for any other purposes than modding IE games.

You can, it's a full blown programming language, and I've done binary patching with it for stuff that was completely unrelated to IE modding more than once. The real issue is, it has too many features you don't need for anything else but IE modding, and dealing with things that are convenient for modding but incovenient for regular programming make it hard to justify using it. It indeed is a niche language and finding uses for it is hard, but it can do a multitude of things that are harder to perform in other languages (byte-editing regexp-matched strings comes to mind, though a good number of non-compiled languages are quite good at this as well, however off the top of my head I wouldn't know how to do it in them while I could put the WeiDU code for that down in less than ten seconds with REPLACE_EVALUATE. I would argue that it's also due to me not using those languages as much, so this isn't as important as I make it sound).

2 hours ago, RoyalProtector said:

Heavens and hells, I'll take C and C++, at least there is more high level stuff there.

That's just because someone went a wrote libraries for those languages. Imagine trying to use C without stdlib. I've done that plenty back when I was learning how to code in C (and WeiDU!) about a decade ago, and I can say, it's pretty a painful experience. WeiDU without using any libraries aside from what is shipped with it is comparable to C++ without any of its standard libraries aside from the C ones. Sadly the case for WeiDU is that you either put on the wizard hat and code out the library yourself or put on some apprentice robes and try to figure out how other people's code works, as there is no real standard library for it out there. The difference between how modders handle code ownership and how that works with GPL'd code in other programming languages also has a pretty big impact, though given that all the people that have written stuff worth using will let you use it if you just ask nicely it isn't much of a problem as long as you're willing to ask them (and learn how to use their code).

[RoyalProtector]

1 hour ago, CrevsDaak said:

That's just because someone went a wrote libraries for those languages

Fair but at least you have tools to encapsulate information and functionality, since that is part of the language as such. Since we're talking about this, are there "libraries" out there for WeiDu (i.e. collections of commonly used functions, or something along those lines)?

It would be a nice project to create a repository of code like that from many knowledgeable contributors.

[lynx]

A few extra functions are shipped with it, but the closest thing to an assortment of other functions and libraries is:

 

[Jarno Mikkola]

And multiple mods have the "lib" folder that have assortments of parts that the they themselves use.

But the greatest: "Things that prevent WeiDU successor or alternative?" is weidu itself. It's better than anything any one has consistently been updating. And the other thing is that dabus burned out.

[DavidW]

It might be helpful for this conversation to have a more precise statement of what WEIDU actually does. Here's mine: WEIDU is a programming language with the following main features:

1) It has a very lightweight implementation, distributable as a self-contained executable small enough to be distributed directly with a mod. (The Windows WEIDU executable is 1.33MB.)
2) It contains a bunch of commands to read files into memory and write them back out to file, all of which have built-in reversibility, i.e. all changes can automatically be uninstalled. That uninstallation is very robust: you can tell WEIDU to make irreversible changes, but it takes active effort and expertise. Normally you can install a WEIDU mod component and be confident that it will uninstall cleanly.
3) Its read/write structure interacts very transparently with the BIF system in IE games: commands like COPY_EXISTING and RESOURCE_CONTAINS will fish files out of chitin.key or the override, as appropriate, without a mod developer needing to pay any attention to the BIF structure. Commands like COPY_EXISTING_REGEXP likewise iterate transparently over game files, irrespective of how they are stored.
4) It provides tools to interact with dialog.tlk (transparently as to whether the game is EE or original, and as to which language is installed). There are several ways it does this - RESOLVE_STR_REF, SAY, STRING_SET_EVALUATE, GET_STRREF, READ_STRREF, and more - and in each case the changes are made reversibly. Many of the tools are transparent: if you are giving names to a magic item, for instance, you don't need to know anything about the underlying .tlk reference.
5) It compiles and decompiles (again, reversibly) BCS scripts to and from the BAF format. In doing so it fairly transparently handles strings embedded in BAF files.
6) It compiles and decompiles (again, reversibly) DLG files to and from WEIDU's own D format. It's worth noting that this is a much more involved process than BAF<->BCS compiling: the D format has a lot of pretty advanced features. And again (much more importantly than for BAF) it transparently handles adding strings in a D file to dialog.tlk.
7) It implements the TRA system for localizing strings.
8 ) It contains a bunch of commands (READ_BYTE/SHORT/LONG/ASCII, GET_OFFSET_ARRAY, etc) that are optimized for interacting with IE-type files at a fairly low level, and (I think) do so quite quickly and efficiently. (You would probably want at least the option of this low-level interaction in any tool: if you want to, say, iterate over every CRE file in the game and get some basic information from it, it's going to be inefficient to do that by reading each one into a data structure.)
9) It integrates quick information retrieval from the various IDS files that the game uses, via IDS_OF_SYMBOL and LOOKUP_IDS_SYMBOL_OF_INT.

(In addition it has a bunch of commands like ADD_SPELL that operate on files at a higher level, but those are inessential: they could be implemented as functions without much loss.)

Let's consider what a WEIDU replacement/successor would require. It would have to be either (i) a new bespoke language written for IE modding, or (ii) an existing language equipped with a robust function library built for IE modding.

(i) seems to me a non-starter. The list above hopefully demonstrates how much work it would be to write that new bespoke language, and while WEIDU is no doubt not what one would design from scratch, I don't see the gains from redesigning it would pay for themselves even setting aside the compatibility issues.

(ii) is more interesting, and I take it closer to what various people in this thread have in mind: using an existing language gives you powerful and robust features like better data-structure handling, variable localization, functional programming, and a big function library, all for free, and also might increase familiarity. (Set aside my skepticism about how much that really matters.) There is a huge backward-compatibility problem, but perhaps that could be finessed by embedding the language in WEIDU: a component would just do AT_NOW (whatever), or more sensibly wrap everything in a function, and the core of the mod would be in the new language. You would either need to require users to install the new language, or else use something small enough that it could be included in the mod itself. The latter seems the better way to go: LUA, for instance, is probably small enough, albeit its footprint is a lot bigger than WEIDU itself.)

I'm still skeptical, for two reasons. Firstly, it would be very difficult to replicate WEIDU's robust uninstallability. Even if one's library of functions for manipulating IE files was carefully built to enforce reversible installation, the user would have native access to LUA, and LUA code itself is under no particular obligation to be reversible. Clean uninstallation would require much more code discipline, rather than happening pretty automatically under WEIDU.

Secondly, and MUCH more importantly, replicating all WEIDU's functionality (as listed above) is a Herculean task. Thousands of hours have gone into WEIDU development and I think thousands of hours would have to go into building a full-featured successor. Implementing the D format alone would be a massive amount of work: it would be tempting to use WEIDU itself to compile and decompile dialogs, but then you would struggle to integrate with dialog.tlk editing. Some time could be saved if the new language was OCAML, so that you could re-use existing code, but I'm not sure replacing WEIDU with a fairly obscure functional-paradigm language is really what people considering a successor have in mind.

What is more plausible - and this now speaks to the exchange between Suy and myself upthread - is to have some bit of LUA (or other) code embedded in WEIDU to carry out some specific task. It would not be difficult to write a WEIDU function with this schematic form

DEFINE_ACTION_FUNCTION lua_execute
    STR_VAR bin="%MOD_FOLDER%/bin" // path to LUA executable
        code="" // path to lua code
        input="" // space-separated list of inputs to the LUA function
BEGIN

    // install LUA if not already installed

    [blah]

    // implement the LUA code, which dumps all its output in the folder weidu_external/workspace/lua_out, and
    // returns variable 'success', equal to 1 if the code successfully implements

    [blah]

    // propagate a LUA failure to a WEIDU failure

    ACTION_IF !(success=1) BEGIN
        FAIL "lua_execute: failed to execute LUA code %code%"
    END

    // sweep LUA output into override (doing it this way lets us use WEIDU uninstall)

    COPY "weidu_external/workspace/lua_out" override
END

But it's important to note the limitations of this: without a LOT of extra work, the LUA code could not interact with dialog.tlk, so strings could not be set; similarly, the code couldn't engage with scripts or dialogs, and would need files to be extracted by WEIDU into override in advance. A tool of this kind would be relatively special-purpose, e.g. a more sophisticated, data-structure-style way to edit item or spell files where changing the .tlk entries isn't needed. I think the jury's out how that would compare to building the tool natively in WEIDU. (Speed is one possible advantage: high-level WEIDU functional methods are sometimes a bit sluggish.) What it wouldn't do is obviate the need to learn WEIDU to write a fully-featured mod.

[mickabouille]

> It provides tools to interact with dialog.tlk

As an aside, is there away to have it update multiple game langage? I mean, keeping theability to change game language at runtime in the game options?

At the moment I still haven't found a way to do it.

[DavidW]

I’m not aware of a way.

[lynx]

I don't get the point of lua_execute above, since you can already do that now with AT_NOW + COPY.

[DavidW]

Higher-level functionality, saves you doing the (OS-dependent) AT_NOW, COPY, and reset directory each time. (After all, in a certain sense there’s no point in any function, since you can do everything with low level commands.)

[lynx]

Why would it be OS-dependent? Surely one would choose a language without major portability issues.

[AL|EN]

@DavidW 

Please forgive my ignorance, I don't get the 'reversible' part. As far as I know, WeiDU reverses its changes simply by making copies of files in various backup folders. So the uninstallation is only a matter of going back to the previous state of the files. I don't see the magic here which would be a source of legitimate complication. But again, I may just don't see oblivious.

[jmerry]

The source of legitimate complication is that other languages don't do that "make backup copies of everything" step by default. The "don't back anything up" state that you have to deliberately invoke with a special modifier in WeiDU is normal in other languages.

[subtledoctor]

31 minutes ago, AL|EN said:

@DavidW 

Please forgive my ignorance, I don't get the 'reversible' part. As far as I know, WeiDU reverses its changes simply by making copies of files in various backup folders. So the uninstallation is only a matter of going back to the previous state of the files. I don't see the magic here which would be a source of legitimate complication. But again, I may just don't see oblivious.

Just that any doofus like me can, with Weidu, get the benefit of that without even trying. Whereas, with another language I would have to intentionally set that up, and 1) hopefully I don’t forget, 2) hopefully it’s not too much extra effort, and 3) hopefully I don’t screw it up.

That’s a lot of hopefully!

Edited March 17 by subtledoctor

[CamDawg]

11 hours ago, mickabouille said:

As an aside, is there away to have it update multiple game langage? I mean, keeping theability to change game language at runtime in the game options?

No, mainly because of the subtle way WeiDU interacts with dialog.tlk.

When a mod uses a new string, WeiDU looks in the existing talk table for a matching string; if it finds a match it uses the existing reference instead of making a new tlk entry. If it doesn't need a new entry in English, but needs one in French, you're already in a conflict since a creature or spell file can only reference one of those values. So you either force the French tlk to use the English reference value or force the English tlk to make a new entry for a redundant string. The former is bad since there's no way to know if the forced French string is a good match, and the latter is bad because if it's a feedback string for a spell (think of something like "level drained" or "held") then everything that might block the string now requires an update--and mods that come after won't know it's there at all. And all of this is a best-case scenario, where you're talking the first mod on a clean install--it will get exponentially worse as you add more mods.

[mickabouille]

Thanks for the write-up but... I re-read it thrice and I'm not sure I understand.

Am I understanding correctly that the basic problem is that the vanilla tlk are not a 1:1 match in different languages?

Or that the correct way for weidu to edit the tlk would be to never reuse existing strings and always create new ones?

 

Both seem more a problem of "didn't think about it at the beginning"

 

EDIT : ah well, the original versions only came with one language IIRC so it probably wasn't even a problem

Edited March 17 by mickabouille