Preprocessing and templating

[Magus]

I'm entertaining an idea to add some preprocessing and/or templating capabilities to MLS/IElib. Looking to gather some feedback before endeavouring.

Gcc was my first thought, but it's not great with loops, so maybe jinja2 instead (or in addition).

Some example snippets to get the ball rolling:

baf

IF
  See(NearestEnemyOf(Myself))
  Global("abi_dalzim","LOCALS",0)
  LevelGT(LastSeenBy(Myself),15)    // Level 16+
  !GlobalTimerNotExpired("delay","LOCALS")
THEN
  RESPONSE #100
    SetGlobalTimer("delay","LOCALS",6)
    IncrementGlobal("doomed","LOCALS",1)
    IncrementGlobal("abi_dalzim","LOCALS",1)
    ReallyForceSpell(NearestEnemyOf(Myself),WIZARD_ABI_DALZIMS_HORRID_WILTING)
END

IF
  See(NearestEnemyOf(Myself))
  Global("efreeti","LOCALS",0)
  LevelGT(LastSeenBy(Myself),15)    // Level 16+
  !GlobalTimerNotExpired("delay","LOCALS")
THEN
  RESPONSE #100
    SetGlobalTimer("delay","LOCALS",6)
    IncrementGlobal("doomed","LOCALS",1)
    IncrementGlobal("efreeti","LOCALS",1)
    ReallyForceSpell(Myself,WIZARD_SUMMON_EFREET)
END

IF
  See(NearestEnemyOf(Myself)
  Global("sunfire","LOCALS",0)
  LevelGT(LastSeenBy(Myself),15)    // Level 16+
  !GlobalTimerNotExpired("delay","LOCALS")
THEN
  RESPONSE #100
    SetGlobalTimer("delay","LOCALS",6)
    IncrementGlobal("doomed","LOCALS",1)
    IncrementGlobal("sunfire","LOCALS",1)
    ReallyForceSpell(NearestEnemyOf(Myself),WIZARD_SUN_FIRE)
END

gcc equivalent

#define HASH1 # // special character shenanigans
#define HASH2(x) x
#define cast_spell_level(SYMBOL, TARGET, SPELL_VAR, MIN_LEVEL) \
  IF \
    See(NearestEnemyOf(Myself)) \
    Global(#SPELL_VAR,"LOCALS",0) \
    LevelGT(LastSeenBy(Myself),MIN_LEVEL) \
    !GlobalTimerNotExpired("delay","LOCALS") \
  THEN \
    RESPONSE HASH2(HASH1)100 \
      SetGlobalTimer("delay","LOCALS",6) \
      IncrementGlobal("doomed","LOCALS",1) \
      IncrementGlobal(#SPELL_VAR,"LOCALS",1) \
      ReallyForceSpell(SYMBOL, TARGET) \
  END

cast_spell_level(WIZARD_ABI_DALZIMS_HORRID_WILTING, NearestEnemyOf(Myself), abi_dalzim, 15)
cast_spell_level(WIZARD_SUMMON_EFREET, Myself, efreeti, 15)
cast_spell_level(WIZARD_SUN_FIRE, Myself, sunfire, 15)

jinja2

{% for item in data.battle_spells %} // list defined elsewhere
IF
  See(NearestEnemyOf(Myself))
  Global("{{ item.var }}","LOCALS",0)
  LevelGT(LastSeenBy(Myself),{{ item.min_level }})
  !GlobalTimerNotExpired("delay","LOCALS")
THEN
  RESPONSE #100
    SetGlobalTimer("delay","LOCALS",6)
    IncrementGlobal("doomed","LOCALS",1)
    IncrementGlobal("{{ item.var }}","LOCALS",1)
    ReallyForceSpell(NearestEnemyOf(Myself), {{ item.symbol }})
END
{% endfor %}

Of course, complex baf is already dominated by SSL, so this is mostly useful in less complex scripts.

But on the other hand, the applications are not limited to baf.

// never again confuse "GLOBAL" and "LOCALS" types
#define LocalGT(x,y) GlobalGT(x, "LOCALS", y)
IF
  LocalGT("my_var", value)
  ...


// just because I'm tired of typing COPY_EXISING_REGEXP all the time
#define copy_er COPY_EXISING_REGEXP
copy_er GLOB ~*.itm~ ~override~


// or take it further
#include header.h
// header.h:
#define POUND1 $ // more shenanigans
#define POUND2(x,y) x ## y
#define POUND3(x,y) POUND2(x,y)
#define POUND_APPEND(x) POUND3(x,POUND1)
#define copy_erg(FILE_EXT) COPY_EXISTING_REGEXP GLOB ~^.+\.POUND_APPEND(FILE_EXT)~ ~override~
// actual code
copy_erg(dlg)
  DECOMPILE_AND_PATCH BEGIN x = 0 END
BUT_ONLY


// this is too long...
IF ~~ study1
  SAY @14 
  IF ~OR(2) Global("wm_book_spell","LOCALS",0) Global("wm_book_spell","LOCALS",1)~ REPLY @20 GOTO 00   // -> +6 = 6/7
  IF ~OR(2) Global("wm_book_spell","LOCALS",0) Global("wm_book_spell","LOCALS",6)~ REPLY @22 GOTO 02   // -> +1 = 1/7
  IF ~OR(2) Global("wm_book_spell","LOCALS",7) Global("wm_book_spell","LOCALS",8)~ REPLY @23 GOTO 03   // -> +2 = 9/10
  IF ~OR(2) Global("wm_book_spell","LOCALS",7) Global("wm_book_spell","LOCALS",9)~ REPLY @24 GOTO 04   // -> +1 = 8/10
// how about this
#define if_node(value1, value2, tra, goto) \
  IF ~OR(2) Global("wm_book_spell","LOCALS",value1) Global("wm_book_spell","LOCALS",value2)~ REPLY tra GOTO goto
IF ~~ study1
  SAY @14
  if_node(0,1,@20,00)
  if_node(0,6,@22,02)
  if_node(7,8,@23,03)
  if_node(7,9,@20,04)

Technically, all this already can be done manually. But with some glue code in various places it could be automated similarly to how SSL works.

 

Edit: no one, really? OK, I guess I'll just do my own thing, as always.

Edited July 10, 2021 by Magus

[suy]

Hi. Thank you for bringing this topic up. I wanted to make public my own party AI script soon (which is mostly inspired by the existing ones, and it's not too long, as I prefer to control almost everything, it just helps with the tedious part). Very early in the development process I noticed the problem of needing replacing (for the DisplayStringHead message), and also I did notice that there are too many repetitive tasks. I knew of SSL, but it's a bit like relying on a moving target without much documentation when I am a noob in scripting anyway.

I ended up using cpp (the C/C++ preprocessor), as it's pretty ubiquitous and I'm familiar with it's syntax quite well. I also looked at other macro languages like M4, but it felt quite crazy. I went for cpp and accepted the repetition because I hate multiple line expansions and the ugly backslash at the end that it would be required for that.

Since now you mentioned text templating systems, I went for Mustache in my case. It's a "logic less" template system with tons of implementations in different languages and a formal specification. It's well stablished. The default/initial Mustache implementation (Ruby based) is a bit dated, but has a command line tool, so you can just define the data in JSON/YAML and the template. You call the tool, and it produces the output, no code involved. However, I needed a more up to date implementation to use the inheritance feature, so I went with a C++ library called bustache. I can easily make a command line tool that just reads the files from input, like SSL does, and produces output. Let me know if anyone is interested. I could even make that into a GUI application, though I don't know if that's any benefit. For my use case, I wrote a program for myself, but again, can easily made abstract so you just need files and don't need to code a thing.

Now, this is how the "code" looks.

Templates that I need to reuse verbatim, just look like the output, so like BAF:

// This is in one file that I've called "ClassesWithStealth"
OR(3)
    Class(Myself,THIEF_ALL)
    Class(Myself,RANGER_ALL)
    Class(Myself,MONK)

// This is another, named "ModalStatesDisabled"
!ModalState(DETECTTRAPS)
!ModalState(STEALTH)
!ModalState(TURNUNDEAD)
!ModalState(BATTLESONG)

The following is a helper that I need to use twice, with a small differences, and which use the previous templates by including them. This is a bit more advanced because uses a feature called "inheritance". The next template I think it's called a "parent", because you can use it like this, or "inherit" it by overriding parts. It's simpler to see it than to explain it, so let's look:

// This line is to change the delimiters. Just because in my keyboard is a pain to write
// {{foo}} (the default in Mustache and which gives it its name), so I'd rather prefer
// to write |foo|. Can be changed to almost anything you like. The next examples don't do it.
{{=| |=}}
IF
    // The notation with the dollar marks a chunk of text with a default value
    // but that can be overridden (in one case I use 0 and no extra BAF triggers, but
    // in another I add a check for enemies, and with a different variable value).
    // See the following file (in the next code block) to see how this is used.
    Global("suy#fight","LOCALS",|$fight|0|/fight|)
    Global("suy#mode","LOCALS",1)
    ActionListEmpty()
    !GlobalTimerNotExpired("suy#hide","LOCALS")
    // Likewise. The default definition on this template has no extra content,
    // but a user can override this, like if it were a function call. See below.
    |$check|
    |/check|
    // This reuses the snippets mentioned above. Just copy them verbatim.
    |>ClassesWithStealth|
    |>ModalStatesDisabled|
THEN
    RESPONSE #100
        SetGlobalTimer("suy#hide","LOCALS",ONE_ROUND)
        Hide()
END

Now with this last template defined in AttemptToHide it can be used in the main template like this:

// Just use the helper template, without overriding anything.
{{<AttemptToHide}}
{{/AttemptToHide}}

// Same helper template, but this time override the defaults to check
// for a different value of the variable, and add an extra line for an
// additional trigger.
{{<AttemptToHide}}
{{$fight}}1{{/fight}}
{{$check}}
    !See([EVILCUTOFF])
{{/check}}
{{/AttemptToHide}}

The main template follows like the next snippet. It looks 100% like regular BAF in the first block, but with the second uses one template (that we've seen above) as "helper", to avoid repetition.

IF
    Global("suy#fight","LOCALS",1)
    Global("suy#mode","LOCALS",1)
    ActionListEmpty()
    !See([EVILCUTOFF])
    Class(Myself,BARD_ALL)
    !ModalState(BATTLESONG)
THEN
    RESPONSE #100
        BattleSong()
END

IF
    Global("suy#fight","LOCALS",1)
    Global("suy#mode","LOCALS",2)
    ActionListEmpty()
    !See([EVILCUTOFF])
    OR(2)
        Class(Myself,THIEF_ALL)
        Class(Myself,MONK)
    {{>ModalStatesDisabled}}
THEN
    RESPONSE #100
        FindTraps()
END

So that's the idea. You write BAF inside a template, but when you see repetitive parts, you move them to files and include them N times. If those need to be reused but with changes, put the default values wrapped in {{$some_name}}default value{{/some_name}}. If you find the double braces too verbose, do as in the example above, and change the delimiters to whatever you like (I did change them to simple vertical bars, as you saw).

I did not need to use loops here, but that's possible as well. Normally the data is defined elsewhere though. I could simulate SSL's target/trigger blocks by having those strings in an array. The template can loop over them, with different values in each iteration.

Conclusions?

I don't have them yet. I just ported my script which was 95% regular BAF, to this templated solution, so I'm still a noob on this approach. It looks a lot clearer in a sense (things have names, and there is no repetition, so I won't go back to writing all BAF by hand, as it's too error prone), but the syntax is too verbose. Having to open and close each use of a different file is too noisy. It should look like a function call, much terser. Given this limited scope (just a simple party AI script), I probably could do writing a program in any language and save snippets in variables and use the formatting capabilities to have default values or their replacement. This seems easy to do in a bunch of languages I have experience with.

I think I can do everything in SSL with Mustache templates, but the number of pros (documented, well implemented) comes with the big, clear con of not being as concise, unless I figure a way how to do it. For example, copying this from SSL lesson six:

IF TRIGGER
TargetBlock(PCMages|PCsInOrderShort)
TriggerBlock(Paralyse|MR)
THEN DO
Action(Spell,WIZARD_HOLD_PERSON)
END

This notation (if I'm not mistaken) would entail having to merge the list of entries of PCMages and PCsInOrderShort somehow (I don't think it can be done at the template level, so it would have to be in some other logic), and likewise for Paralyse and MR. Then, make a nested loop so each combination of target+trigger is looped over, and expand the action. SSL wins hands down in being somewhat simpler once you've learnt it.

Note that I've not written anything with SSL yet, but I'll try to write my party AI script with it to compare side by side, though it will take time, and I'm not even sure it will be easier for this purpose.

Cheers, and a cookie for you if you read the wall of text.

[Magus]

Well, it's probably a no-brainer for a c++ programmer, but as for me, I don't find the syntax too intuitive. My jinja2 example above seems much clearer for casual reader.

Also not sure if I like the logic-less approach, could be too limiting for this case.

Of course, there's no reason why diffferent engines can't be used. Or even combined. But highlighting and intellisense become a bitch in that case.

In any case, I wouldn't go for hardcoding snippets. Rather, I'd publish a library to let anyone who want to use it include it.

[suy]

10 minutes ago, Magus said:

Well, it's probably a no-brainer for a c++ programmer, but as for me, I don't find the syntax too intuitive.

Well, I don't find it intuitive either. The Mustache initial implementation was Ruby, and a lot of support and proposals for improvements came from Twitter folks doing JavaScript, I think. C++ is not even a language in which is common to use template systems because it's not popular for making web pages. All the other template systems that I've seen have the problem that are their own language. So they might be intuitive initially, but you have to learn them anyway to write code. The logic-less approach may not make complete sense for generating code, I give you that. But in that case, I'd rather go with a template system that is the very opposite of this, like the one in underscore/lodash, based on John Resig's approach. That allows you to use the real language.

27 minutes ago, Magus said:

But highlighting and intellisense become a bitch in that case.

But do you have code completion, validation and so on on the language server for BAF or BAF+jinja2? I definitely have to check it out by myself, but I just don't use VS Code and I can't switch so easily.

My examples, are just the very proof of concept, and the first use I've done with this, ever. I've written very little BAF code so far. But it would not be hardcoded, as all would be files that you supplement the main template with, and you can provide as many or as few as you want.

[Magus]

22 minutes ago, suy said:

But do you have code completion, validation and so on on the language server for BAF or BAF+jinja2? I definitely have to check it out by myself, but I just don't use VS Code and I can't switch so easily.

Switch from what, notepad++? Or do you write BAF in some IDE that has no support for it at all?

Yes, MLS has some support for BAF. It's very basic, but better than nothing.

If and when there's some consensus about templating for weidu (which preferably shouldn't be limited to BAF), then extending MLS for supporting it might be considered.

[suy]

I do most of my editing in Vim (now NeoVim). I wrote the syntax highlighting for BAF myself because I could not find one online for it. 5 years ago I had to also start using an IDE for C++ development because the tooling wasn't there, and there was no LSP (and in the IDE I still use vi-mode). Now I'm considering moving that part of development to NeoVim, but I need time to invest on it. MLS is another reason to start learning about using LSP on this editor. Though, to be honest, I probably will never write stuff in WeiDU. But I still want to read code from existing WeiDU mods.

[Magus]

Oh, I see you're a man of culture as well.

[suy]

On 8/7/2021 at 10:50 PM, suy said:

Though, to be honest, I probably will never write stuff in WeiDU. But I still want to read code from existing WeiDU mods.

Well, this aged poorly, @suy from the past. I've actually written WeiDU since then.

Anyway, I want to add a followup to this thread, because I've done another step in this direction. Or to the parallel of this direction, because I dropped Mustache, and I've gone with a library in Lua that implements templates like the approach I mentioned in this thread: embedding the real programming language in the template. It looks like this for a simple example:

<% for n = 1, 6 do -%>
IF
    StateCheck(Player<%- n -%>, STATE_INVISIBLE)
    CheckStatGT(Player<%- n -%>, 1, BACKSTABDAMAGEMULTIPLIER)
THEN
    RESPONSE #100
        ApplySpellRES("S_FSTAB",Player<%- n -%>)
        Continue()
END

<% end -%>

That is a trivial enough example that just repeats that block 6 times for the 6 players. Not a big deal to do by hand because is so simple (and not worth using a template for this), but in the case of my party AI script is, among other things, 2 attack blocks per Neareast/SecondNearest/etc which are very repetitive, and each block is significantly larger than this, so it was hellish to maintain when trying to do changes. When I was experimenting I definitely needed to have a small program generate the BAF code, or I would have turned mad. Stripping whitespace and comments, the templates are about 225 lines, and the resulting output is 425 lines. Almost twice the size for the current version. But the savings can be much bigger than this.

I'm processing the template on my computer, and adding the output to the Git repository, so the user doesn't run any Lua code. But if I were to add Lua to the mod, the footprint would be still very small. And exists a version of Lua that is a single executable that works on either Linux/Windows/Mac. I will try this eventually, I think.

If anyone is interested in this, I can elaborate a lot further, with simple examples for non-programmers, if I was too technical. But the code required for this is all now available in Gitlab: https://gitlab.com/moebiusproject/all-the-things-revised/-/tree/27fdf386dfb61ac4c8d6184426c083d9b6da633a/all-the-things-revised/templates

Another example of how it looks (same-ish as the example from some years ago):

<%
fight = fight or 0
check = check or ''
%>
IF
    Global("suy#fight","LOCALS",<%-fight-%>)
    Global("suy#mode","LOCALS",1)
    ActionListEmpty()
    !GlobalTimerNotExpired("suy#hide","LOCALS")
<%- check %>
<%- render("ClassesWithStealth.etlua") -%>
<%- render("ModalStatesDisabled.etlua") -%>
THEN
    RESPONSE #100
        SetGlobalTimer("suy#hide","LOCALS",ONE_ROUND)
        Hide()
END

The 4 lines at the top look a bit... weird, perhaps, but they could be removed, and instead you'd have:

// instead of this (assumes "fight" has a value)
Global("suy#fight","LOCALS",<%-fight-%>)
// ...you would write this (if "fight" has a value, that, otherwise 0)
Global("suy#fight","LOCALS",<%-fight or 0-%>)

And the above template would be used like this in another template:

<%- render("AttemptToHide.etlua", {
        fight=1,
        check='    !See([EVILCUTOFF])',
    }) -%>

I think this can accomplish more or less the same than Stratagems Scripting Language, but this is a much simpler implementation, at the cost of a less integrated feeling in how the script looks. It's clearly a syntax embedded into another, instead of an evolved syntax. With a bit of extra work I think I can iron out a bit the lack of syntax sugar, but is never going to be the same (sometimes for worse, but sometimes for better IMHO).

[lynx]

Seems like you reimplemented a subset of what liquid and other templating solutions offer — seems odd.

[suy]

No, I did the very opposite. I used a library that, as I explained, doesn't invent a new language at all, but uses the "host" language. The etlua library is 285 lines of a single Moonscript file that compiles to 400 lines of pure Lua. Liquid seems to be 3700 in the 'lib' directory. The comparison is not to Liquid, but to ERB (which seems to be 486 lines in the 'lib' directory). In the other post I mentioned John Resig's microtemplate, and Lodash/Underscore, which are also well known, and it's the same approach.

Any of those implementations would work just fine. I like ERB and Ruby, and I like those JS libraries as well, and they would work fine. Lua is a bit better in that is much smaller, and it's somewhat known to other modders, as it's used in EEex, and the UI.

[Magus]

Revisiting this, I think that templating is fine for personal use, but it's going to be hard to find a solution that's liked widely.

To really step up the game, I'd rather have a complete transpilation solution from another language, preferably with strict typing. Probably not for weidu code, but for d and baf I can see it working.

The choice would be basically between Python and Typescript. Well, I guess Lua could also be considered. But I'm leaning towards Typescript. Not that I like it too much, but the tooling is so much better.

[suy]

I've considered this as well. I wish I could write nested "if" clauses (and "else" clauses) on BAF code, of course. So I thought of a custom language that would look like a BAF superset, but with those extras (and it would be similar to SSL in pros/cons and philosophy). I just don't see how to do this properly, given that it would look so very differently once transpiled, that debugging the generated code would be difficult given the lack of support for any BAF debugging, source maps, etc. Also, writing a compiler is not in my comfort zone or interests.

But definitely interested in looking up what others can come up with.

So far, after trying the C preprocessor and Mustache, and now this Lua templates, I think I've found my sweet spot. I will try to make an additional post in the future, once I've ironed out some details, because I think this can be a sweet spot for other people too, if I can provide good examples and ease of use. There are other language and template choices, though, but I think any could do well enough.