A Course in WEIDU

[DavidW]

2 minutes ago, Luke said:

7 hours ago, DavidW said:

Are you suggesting I should put it in the document? I could do, but it's already 60 pages long and that's with a fairly active attempt to stay focused on the essentials.

Yes, it might be useful to know this trick...

Sure... but there are hundreds of WEIDU tricks it might be useful to know. A core requirement in writing any kind of introduction is discipline about keeping its length under control.

4 minutes ago, Luke said:

7 hours ago, DavidW said:

If you really do want to optimize for speed, I would lose all those PATCH_WITH_SCOPEs - a quick test gets nearly a 50% speedup without them. You need to do PATCH_CLEAR_ARRAY fx_data each time you loop through a new ability...

Are you sure about this...?

Asking because it takes roughly 1 second more than before for me  (as far WeiDU Timings are concerned, the most relevant difference between having all those PATCH_WITH_SCOPEs and not having them is `process_patch2` => 2.199 with PATCH_WITH_SCOPEs vs. 3.057 without PATCH_WITH_SCOPEs...)

Your code takes about 3 seconds on my computer; that drops to a bit under 2 seconds if I take out the PATCH_WITH_SCOPEs. (WEIDU timings are not always consistent, especially for short timings; it's worth running it a few times and comparing.)

[Luke]

1 minute ago, DavidW said:

A core requirement in writing any kind of introduction is discipline about keeping its length under control.

Fair enough...

1 minute ago, DavidW said:

Your code takes about 3 seconds on my computer; that drops to a bit under 2 seconds if I take out the PATCH_WITH_SCOPEs. (WEIDU timings are not always consistent, especially for short timings; it's worth running it a few times and comparing.)

Just to clarify: is this the code without all those PATCH_WITH_SCOPEs...?

Spoiler

ACTION_TIME "clone_effect_ex" BEGIN
	WITH_SCOPE BEGIN
		COPY_EXISTING_REGEXP "^.+\.\(spl\|itm\)$" "override"
			PATCH_MATCH "%DEST_EXT%" WITH
				"SPL" BEGIN
					GET_OFFSET_ARRAY "ab_array" SPL_V10_HEADERS
					SET "ab_size" = 0x28
				END
				"ITM" BEGIN
					GET_OFFSET_ARRAY "ab_array" ITM_V10_HEADERS
					SET "ab_size" = 0x38
				END
				DEFAULT
					PATCH_FAIL "Should not happen (~%DEST_FILE%~)"
			END
			PHP_EACH "ab_array" AS "ab_ind" => "ab_off" BEGIN
				PATCH_CLEAR_ARRAY "fx_data"
				GET_OFFSET_ARRAY2 "fx_array" "%ab_off%" SPL_V10_HEAD_EFFECTS
				PHP_EACH "fx_array" AS "fx_ind" => "fx_off" BEGIN
					PATCH_IF (LONG_AT ("%fx_off%" + 0x24) BAND (BIT0 BOR BIT1 BOR BIT2 BOR BIT3 BOR BIT4)) BEGIN
						READ_ASCII ("%fx_off%" + 0x0) $"fx_data"("%fx_off%") (0x30)
					END
				END
				// Actual cloning
				SET "fx_count" = 1 // this variable is needed to take into account the fact that data moves with each iteration – set it to 0 for "insert"="above"
				PHP_EACH "fx_data" AS "fx_off" => "fx_attributes" BEGIN
					INSERT_BYTES ("%fx_off%" + 0x30 * "%fx_count%") 0x30
					WRITE_ASCIIE ("%fx_off%" + 0x30 * "%fx_count%") "%fx_attributes%"
					WRITE_LONG ("%fx_off%" + 0x30 * "%fx_count%" + 0x28) (STHIS - 2) // update the `savebonus` field
					SET "fx_count" += 1
					WRITE_SHORT ("%ab_off%" + 0x1E) (THIS + 1) // update # effects
					// Update `1st_effect_idx` on all subsequent abilities
					FOR ("i" = 1 ; "%i%" < SHORT_AT 0x68 - "%ab_ind%" ; "i" += 1) BEGIN
						WRITE_SHORT ("%ab_off%" + "%i%" * "%ab_size%" + 0x20) (THIS + 1)
					END
				END
			END
		BUT_ONLY_IF_IT_CHANGES
	END
END

 

Also, why is PATCH_WITH_SCOPE supposed to be so relevant...? I mean, it's basically a function that takes no argument and returns nothing...

[Graion Dilach]

It provides sandboxing, which even functions defining/touching globals inside them doesn't do.

[DavidW]

37 minutes ago, Luke said:

Just to clarify: is this the code without all those PATCH_WITH_SCOPEs...?

Yep.

37 minutes ago, Luke said:

Also, why is PATCH_WITH_SCOPE supposed to be so relevant...? I mean, it's basically a function that takes no argument and returns nothing...

I don't know, but there's sometimes a not-completely-trivial cost in running a function too. I guess WEIDU has to isolate a chunk of memory and manage the local variables, and then get rid of them at the end?... I know basically nothing about implementation architecture. I just noticed that the code was enacting 9,000-odd PATCH_WITH_SCOPEs and wondered if it would make a difference to implementation speed, since they're not needed.

 

13 minutes ago, Graion Dilach said:

It provides sandboxing, which even functions defining/touching globals inside them doesn't do.

I'm not sure that's right, but what did you have in mind?

[Graion Dilach]

The WeiDU doc claims that during exiting from a WITH_SCOPE, even changes made to globals within the scope are reverted. So it basically duplicates the then-current state to a sandbox at start and drops that altogether during exit. I can imagine that being expensive.

Edited July 26, 2022 by Graion Dilach

[Luke]

6 minutes ago, Graion Dilach said:

The WeiDU doc claims that during exiting from a WITH_SCOPE, even changes made to globals within the scope are reverted. So it basically duplicates the then-current state to a sandbox at start and drops that altogether during exit. I can imagine that being expensive.

OK, but then why does it (the no PATCH_WITH_SCOPE variant) take more time in my case...? I tried running it several times and it always took more time (~ 1 second) than the one with all those PATCH_WITH_SCOPEs...

Edited July 26, 2022 by Luke

[DavidW]

17 minutes ago, Graion Dilach said:

The WeiDU doc claims that during exiting from a WITH_SCOPE, even changes made to globals within the scope are reverted. So it basically duplicates the then-current state to a sandbox at start and drops that altogether during exit. I can imagine that being expensive.

That's true for functions too. You can change the value of a global inside a WEIDU function and it gets reverted when the function exists, unless you explicitly return it.

11 minutes ago, Luke said:

OK, but then why does it (the no PATCH_WITH_SCOPE variant) take more time in my case...?

Not a clue.

[Luke]

Well, guess we should also ask @argent77, @Wisp 

Which of the following code snippets is supposed to take less time?

Spoiler

ACTION_TIME "clone_effect_ex" BEGIN
	WITH_SCOPE BEGIN
		COPY_EXISTING_REGEXP "^.+\.\(spl\|itm\)$" "override"
			PATCH_WITH_SCOPE BEGIN
				PATCH_MATCH "%DEST_EXT%" WITH
					"SPL" BEGIN
						GET_OFFSET_ARRAY "ab_array" SPL_V10_HEADERS
						SET "ab_size" = 0x28
					END
					"ITM" BEGIN
						GET_OFFSET_ARRAY "ab_array" ITM_V10_HEADERS
						SET "ab_size" = 0x38
					END
					DEFAULT
						PATCH_FAIL "Should not happen (~%DEST_FILE%~)"
				END
				PHP_EACH "ab_array" AS "ab_ind" => "ab_off" BEGIN
					PATCH_WITH_SCOPE BEGIN
						GET_OFFSET_ARRAY2 "fx_array" "%ab_off%" SPL_V10_HEAD_EFFECTS
						PHP_EACH "fx_array" AS "fx_ind" => "fx_off" BEGIN
							PATCH_IF (LONG_AT ("%fx_off%" + 0x24) BAND (BIT0 BOR BIT1 BOR BIT2 BOR BIT3 BOR BIT4)) BEGIN
								READ_ASCII ("%fx_off%" + 0x0) $"fx_data"("%fx_off%") (0x30)
							END
						END
						// Actual cloning
						PATCH_WITH_SCOPE BEGIN
							SET "fx_count" = 1 // this variable is needed to take into account the fact that data moves with each iteration – set it to 0 for "insert"="above"
							PHP_EACH "fx_data" AS "fx_off" => "fx_attributes" BEGIN
								INSERT_BYTES ("%fx_off%" + 0x30 * "%fx_count%") 0x30
								WRITE_ASCIIE ("%fx_off%" + 0x30 * "%fx_count%") "%fx_attributes%"
								WRITE_LONG ("%fx_off%" + 0x30 * "%fx_count%" + 0x28) (STHIS - 2) // update the `savebonus` field
								SET "fx_count" += 1
								WRITE_SHORT ("%ab_off%" + 0x1E) (THIS + 1) // update # effects
								// Update `1st_effect_idx` on all subsequent abilities
								PATCH_WITH_SCOPE BEGIN
									FOR ("i" = 1 ; "%i%" < SHORT_AT 0x68 - "%ab_ind%" ; "i" += 1) BEGIN
										WRITE_SHORT ("%ab_off%" + "%i%" * "%ab_size%" + 0x20) (THIS + 1)
									END
								END
							END
						END
					END
				END
			END
		BUT_ONLY_IF_IT_CHANGES
	END
END
                                                                   
// VERSUS
                                                                   
ACTION_TIME "clone_effect_ex" BEGIN
	WITH_SCOPE BEGIN
		COPY_EXISTING_REGEXP "^.+\.\(spl\|itm\)$" "override"
			PATCH_MATCH "%DEST_EXT%" WITH
				"SPL" BEGIN
					GET_OFFSET_ARRAY "ab_array" SPL_V10_HEADERS
					SET "ab_size" = 0x28
				END
				"ITM" BEGIN
					GET_OFFSET_ARRAY "ab_array" ITM_V10_HEADERS
					SET "ab_size" = 0x38
				END
				DEFAULT
					PATCH_FAIL "Should not happen (~%DEST_FILE%~)"
			END
			PHP_EACH "ab_array" AS "ab_ind" => "ab_off" BEGIN
				PATCH_CLEAR_ARRAY "fx_data"
				GET_OFFSET_ARRAY2 "fx_array" "%ab_off%" SPL_V10_HEAD_EFFECTS
				PHP_EACH "fx_array" AS "fx_ind" => "fx_off" BEGIN
					PATCH_IF (LONG_AT ("%fx_off%" + 0x24) BAND (BIT0 BOR BIT1 BOR BIT2 BOR BIT3 BOR BIT4)) BEGIN
						READ_ASCII ("%fx_off%" + 0x0) $"fx_data"("%fx_off%") (0x30)
					END
				END
				// Actual cloning
				SET "fx_count" = 1 // this variable is needed to take into account the fact that data moves with each iteration – set it to 0 for "insert"="above"
				PHP_EACH "fx_data" AS "fx_off" => "fx_attributes" BEGIN
					INSERT_BYTES ("%fx_off%" + 0x30 * "%fx_count%") 0x30
					WRITE_ASCIIE ("%fx_off%" + 0x30 * "%fx_count%") "%fx_attributes%"
					WRITE_LONG ("%fx_off%" + 0x30 * "%fx_count%" + 0x28) (STHIS - 2) // update the `savebonus` field
					SET "fx_count" += 1
					WRITE_SHORT ("%ab_off%" + 0x1E) (THIS + 1) // update # effects
					// Update `1st_effect_idx` on all subsequent abilities
					FOR ("i" = 1 ; "%i%" < SHORT_AT 0x68 - "%ab_ind%" ; "i" += 1) BEGIN
						WRITE_SHORT ("%ab_off%" + "%i%" * "%ab_size%" + 0x20) (THIS + 1)
					END
				END
			END
		BUT_ONLY_IF_IT_CHANGES
	END
END

 

In my case it is the first one, whereas on DavidW's computer is the second one... My computer runs macOS and is quite ancient (mid 2012, sigh), whereas DavidW's computer runs Windows (if I recall correctly) and it is probably not that ancient... I'm not sure if it can make a difference, but just so that you know it...

[DavidW]

Put the PATCH_CLEAR_ARRAY into your WITH_SCOPE version and see what that does.

[RoyalProtector]

This is amazing, thank you so much for the guide. I'm currently making a kit revision mod and this will be undoubtedly useful. Thanks again

[suy]

Some small pieces of feedback from someone who started doing a bit of real WeiDU just after you published the doc, and now it's doing some more (on his way to make an actual mod maybe):

• The part about AUTO_EVAL_STRINGS is not explained, and IMHO, with the documentation from WeiDU close to not saying anything meaningful to a learning, it's too problematic to understand that it might be very likely needed. I got hit by wanting to pass a string to a custom function, and using exactly the same syntax as in one of the snippets, it wasn't working. The AUTO_EVAL_STRINGS fixed it because it was suggested to me. Then with a bit of imagination the WeiDU docs started to make a tiny bit of sense (but still not much).
• Scope of variables seem also completely missing from the course and almost missing from WeiDU's docs. That's a massively important topic to me! Everything looks kind of global by default, which is convenient but scary. Are the variables defined to iterate on loops global as well? All of this is a pity that it's not covered, as most languages do not use globals that often. I'd set aside performance considerations, though, specially given the above comments, where it seems the results are not easy to reproduce. Changes in WeiDU's implementation could change performance from version to version significantly, also. I don't know about OCaml, but I know of a few data structures in C++ that do copy on write or structural sharing, and would be a non-issue for copying scopes (in terms of performance). I suppose this techniques are evolving often in functional languages.
• When to use ACTION/OUTER/PATCH_SOMETHING is an incredible mess to me (I suppose one gets used to in the end). I'm making some notes, but I think I'll end up making a table to print and set on the wall. I'll try to make it in some format that you might paste at the end of the course, or attach it here in someone else wants it, because I think a kind of "reference" is necessary. Likewise for the setting and interpolating of variables. Once I can make some sense of it myself, I try to put it down to have it as the typical beginner's "cheat sheet".

[CamDawg]

37 minutes ago, suy said:

Scope of variables seem also completely missing from the course and almost missing from WeiDU's docs.

It wasn't until the introduction of functions that WeiDU even had variable scope. Assume global unless you're explicitly operating under a function or WITH_SCOPE. And I know this will horrify all of the real programmers here, but most of the time limiting scope is not worth the effort.

37 minutes ago, suy said:

When to use ACTION/OUTER/PATCH_SOMETHING is an incredible mess to me

For me, I find the easiest to frame this mentally is to think in terms of actions vs. patches. Many things have both a patch and action version; actions tend to be ACTION_ or OUTER_ and patch are INNER_ or PATCH_ . Granted, WeiDU doesn't help with syntax like OUTER_INNER_PATCH or INNER_ACTION, but it's mainly a matter of keeping track of which mode you're in at the moment. Most of the problems are that WeiDU doesn't always have clear delineation of when you're switching from one context to the other.

Edited November 8, 2022 by CamDawg

[suy]

Thanks. I kinda keep well the context in which I'm in, but being new to the language, I'm totally missing when the keyword has a prefix, the other, or none at all. Like, for the usual "if" there is ACTION_IF and PATCH_IF, but for the usual "for", there is OUTER_FOR and FOR. That's terrible for my awful memory. So I have a scrap of paper by my side right now...

[Graion Dilach]

I almost always have the WeiDU doc opened in a browser tab exactly to look at the same links Cam points at - while I can track that I'm in a patch or in an action, I still tend to Ctrl-F in the lists to look up the appropriate variant. It's no biggie IMO.

Edited November 8, 2022 by Graion Dilach

[suy]

32 minutes ago, CamDawg said:

And I know this will horrify all of the real programmers here, but most of the time limiting scope is not worth the effort.

If WeiDU doesn't make it easy, certainly not. So far I'm ignoring it as much as I can. I wasn't even going to reply to this, but I just got hurt by this right now... I was looping over all the weapon proficiency IDs (bastard sword to sling):

OUTER_FOR (proficiency = bastardSwordProficiencyId; proficiency <= slingProficiencyId; ++proficiency) BEGIN
    OUTER_FOR (value = 1; value <= 3; ++value) BEGIN
        // (...)
        LPF ALTER_EFFECT INT_VAR match_opcode = 233
            parameter1 = %value% parameter2 = %proficiency%
        END
    END
END

 then below adding an extra block for the club, which is some positions below. So I copied and pasted, adjusting accordingly:

OUTER_FOR (value = 1; value <= 3; ++value) BEGIN
    // (...)
    LPF ALTER_EFFECT INT_VAR match_opcode = 233
        parameter1 = %value% parameter2 = %proficiency%
    END
END

But I accidentally left the use of "proficiency" unchanged, so it picked up the value of the previous loop. This is typically an error in most languages because the introduced variable on the loop would be local to said loop. Here it just runs without errors, but it doesn't do the right thing. So I think I'll have to consider WITH_SCOPE for future use.