A Course in WEIDU

CamDawg · June 7, 2022

5 minutes ago, DavidW said:

If you are using it, my suggestion is to back up to weidu_external/backup/[my_mod], just to control clutter. But of course if you just create a modder-prefixed folder in weidu_external and do everything in there, everything will work fine.

I've been using weidu_external/mod_folder, and then organizing within that, e.g. /backup lies within. While it's unlikely I'm going to run into any conflicts in a mutual backup or workspace directory, I can reduce the chance to zero with my own directory.

5 minutes ago, DavidW said:

The way I set up weidu_external, you should be able to do all of this in weidu_external/workspace - the assumption is you should only ever use that subfolder for fire-and-forget on-the-fly creation, so it doesn't need a modder prefix.

It's a mix--some of it (Friendly Random Drops) is throwaway, while something like the 2H Bastard Swords stores (copy w/o backup) the file name mappings for future re-installs. Regardless, it all gets dumped in w_e/cdtweaks.

DavidW · June 7, 2022

6 minutes ago, CamDawg said:

I've been using weidu_external/mod_folder, and then organizing within that, e.g. /backup lies within. While it's unlikely I'm going to run into any conflicts in a mutual backup or workspace directory, I can reduce the chance to zero with my own directory

Perfectly fair.

AL|EN · June 8, 2022

15 hours ago, CamDawg said:

AL|EN's proposed alternative method--which I'm slowly adopting as I update stuff--places the converted tra files outside of the mod folder, avoiding this issue as well.

Tweaks does this, but a simpler example is the recently-shipped SP Collection. I don't want to hijack David's thread with a semi-tutorial but the process is basically:

Ship all tra files as utf-8

On the originals, run the converter to turn them into ANSI/other charset as needed, setting the output somewhere in weidu_external

Set a variable with the path to the tra files--the original files (EE) or the output path from the converter (weidu_external/whatever) on the originals

Change your AUTO_TRA or any other place you're pointing to a tra file to use the path variable

14 hours ago, DavidW said:

It's my method originally (since 2013) - see this thread.

Originally it required a bit of hackery, because HANDLE_CHARSETS didn't support choosing a directory. Wisp added it in v247 to accommodate my encapsulation goal, and Alien did a very nice tutorial.

For the 'course in WEIDU' I was inclined to leave multiplatform installation out just on scope-management grounds, but that may have been a mistake.

It's not quite the same method when it comes to the origin and outcome. My originated from the troubles of dealing with different encoding of the localized files and extra work required during translation updates. The outcome is not only immutability but also the ability to translate files directly using GitHub web editor. IMHO, the UTF8 variant should be default approach when it comes for multi-edition mods (classic+ee) but I agree that it's out of the scope of this guide.

AL|EN · June 8, 2022

About the guide:

- the weidu download link is wrong, should be: https://github.com/WeiDUorg/weidu/releases/latest

(I will post more if I find anything )

Luke · June 8, 2022

@DavidW

When talking about clearing arrays, you might want to share this function.

That is: if possible, you should never use VARIABLE_IS_SET to check if an array element is set

Spoiler

// might be a problem if there exists a variable named "myarray_Melissan"
ACTION_IF VARIABLE_IS_SET $myarray("Melissan") BEGIN
	PRINT "Melissan’s value is set"
END

// safe approach
LAF array_contains
	STR_VAR array="myarray"
			key="Melissan"
	RET value
END
ACTION_IF value BEGIN
	PRINT "Melissan’s value is set"
END

DavidW · June 8, 2022

2 hours ago, AL|EN said:

It's not quite the same method when it comes to the origin and outcome. My originated from the troubles of dealing with different encoding of the localized files and extra work required during translation updates. The outcome is not only immutability but also the ability to translate files directly using GitHub web editor.

I should look more carefully then. (You link to it in my immutability thread and say there that it's an implementation of my method: I took you at your word )

2 hours ago, AL|EN said:

the weidu download link is wrong, should be: https://github.com/WeiDUorg/weidu/releases/latest

It's not wrong: the link I give connects to the github page. (I prefer the biggdu link because you can get more conveniently at the changelog and readme.)

55 minutes ago, Luke said:

When talking about clearing arrays, you might want to share this function.

That is: if possible, you should never use VARIABLE_IS_SET to check if an array element is set

I think that's overkill: indeed VARIABLE_IS_SET needs to be used with caution (I wouldn't normally put it in a library function, for instance) but in many applications it's in practice safe, and it's significantly faster. (As for the guide, I don't want to discuss functions at that point in the guide since they haven't been introduced yet, and I don't want to have people having to rely on my function library in it.)

Luke · June 8, 2022

4 hours ago, DavidW said:

I think that's overkill: ... (As for the guide, I don't want to discuss functions at that point in the guide since they haven't been introduced yet, and I don't want to have people having to rely on my function library in it.)

As you wish.

Having said that, I think it's worth at least mentioning it...

4 hours ago, DavidW said:

... and it's significantly faster.

According to your own tests, the concern about performance is overrated...

DavidW · June 8, 2022

7 minutes ago, Luke said:

Having said that, I think it's worth at least mentioning it...

As I say, I think it’s bad pedagogy.

8 minutes ago, Luke said:

According to your own tests, the concern about performance is overrated...

I meant ‘faster to code’ (and easier to read). (If WEIDU supported non-procedural function calls, it would be different.)

Salk · June 9, 2022

Kudos, DavidW!

I will love reading through this. Who knows, I might learn to do something before it is too late...

Magus · June 9, 2022

Not everything is directly related to coding, but I'll leave it here as a reminder:

https://ielib.bgforge.net/

https://golem.bgforge.net/

https://hive.bgforge.net/

https://bgforge.net/happy-ie/

mickabouille · June 9, 2022

Wow, is that actual continuous integration for mods?

Edited June 9, 2022 by mickabouille

Awachi · June 9, 2022

This is cool.

Luke · July 25, 2022

As far as "Adding and subtracting headers" (page 59) is concerned:

as far as subtracting is concerned, you showed us a handy way to achieve it (the 999 trick):

Spoiler

/* Delete all SPL abilities whose `min_level` is strictly greater than 1 */
ACTION_TIME "delete_spl_ability_ex" BEGIN
	WITH_SCOPE BEGIN
		COPY_EXISTING_REGEXP "^.+\.spl$" "override"
			PATCH_WITH_SCOPE BEGIN
				GET_OFFSET_ARRAY "ab_array" SPL_V10_HEADERS
				PHP_EACH "ab_array" AS "ab_ind" => "ab_off" BEGIN
					PATCH_IF (SHORT_AT ("%ab_off%" + 0x10) > 1) BEGIN
						WRITE_BYTE ("%ab_off%" + 0x0) 0xFF // mark it for later deletion – we can't use 999 here because the `header_type` field is just 1-byte long... As a result, we'll use 0xFF (255, maximum unsigned byte)
					END
				END
				// Actual deletion
				LPF "DELETE_SPELL_HEADER" INT_VAR "header_type" = 0xFF END
			END
		BUT_ONLY_IF_IT_CHANGES
	END
END

/* Delete all global effects (except for op0, 144, 145) from armors */
ACTION_TIME "delete_itm_eqeffect_ex" BEGIN
	WITH_SCOPE BEGIN
		COPY_EXISTING_REGEXP "^.+\.itm$" "override"
			PATCH_IF (SHORT_AT 0x1C == IDS_OF_SYMBOL ("ITEMCAT" "ARMOR")) BEGIN
				PATCH_WITH_SCOPE BEGIN
					GET_OFFSET_ARRAY "fx_array" ITM_V10_GEN_EFFECTS
					PHP_EACH "fx_array" AS "fx_ind" => "fx_off" BEGIN
						READ_SHORT "%fx_off%" "current_effectID"
						PATCH_IF ("%current_effectID%" STRING_MATCHES_REGEXP "^\(0\|144\|145\)$") BEGIN
							WRITE_SHORT "%fx_off%" 999 // mark it for later deletion
						END
					END
					// Actual deletion
					LPF "DELETE_ITEM_EQEFFECT" INT_VAR "opcode_to_delete" = 999 END
				END
			END
		BUT_ONLY_IF_IT_CHANGES
	END
END

As far as adding is concerned, it might be convenient to use WeiDU’s low-level functionality instead of WeiDU's built-in functions (mainly for efficiency – the following piece of code should take less than 5 seconds to patch 3083 files (unmodded IWD:EE)...)

Spoiler

/*
Clone all SPL/ITM V10_HEAD_EFFECTS that offers at least one saving throw. In particular:
- Decrease the clone's `savebonus` field by 2
- Insert the cloned effect immediately below the matched effect
*/
ACTION_TIME "clone_effect_ex" BEGIN
	WITH_SCOPE BEGIN
		COPY_EXISTING_REGEXP "^.+\.\(spl\|itm\)$" "override"
			PATCH_WITH_SCOPE BEGIN
				PATCH_MATCH "%DEST_EXT%" WITH
					"SPL" BEGIN
						GET_OFFSET_ARRAY "ab_array" SPL_V10_HEADERS
						SET "ab_size" = 0x28
					END
					"ITM" BEGIN
						GET_OFFSET_ARRAY "ab_array" ITM_V10_HEADERS
						SET "ab_size" = 0x38
					END
					DEFAULT
						PATCH_FAIL "Should not happen (~%DEST_FILE%~)"
				END
				PHP_EACH "ab_array" AS "ab_ind" => "ab_off" BEGIN
					PATCH_WITH_SCOPE BEGIN
						GET_OFFSET_ARRAY2 "fx_array" "%ab_off%" SPL_V10_HEAD_EFFECTS // NB, same for spl and itm
						PHP_EACH "fx_array" AS "fx_ind" => "fx_off" BEGIN
							PATCH_IF (LONG_AT ("%fx_off%" + 0x24) BAND (BIT0 BOR BIT1 BOR BIT2 BOR BIT3 BOR BIT4)) BEGIN
								READ_ASCII ("%fx_off%" + 0x0) $"fx_data"("%fx_off%") (0x30)
							END
						END
						// Actual cloning
						PATCH_WITH_SCOPE BEGIN
							SET "fx_count" = 1 // this variable is needed to take into account the fact that data moves with each iteration – set it to 0 for "insert"="above"
							PHP_EACH "fx_data" AS "fx_off" => "fx_attributes" BEGIN
								INSERT_BYTES ("%fx_off%" + 0x30 * "%fx_count%") 0x30
								WRITE_ASCIIE ("%fx_off%" + 0x30 * "%fx_count%") "%fx_attributes%"
								WRITE_LONG ("%fx_off%" + 0x30 * "%fx_count%" + 0x28) (STHIS - 2) // update the `savebonus` field
								SET "fx_count" += 1
								WRITE_SHORT ("%ab_off%" + 0x1E) (THIS + 1) // update # effects
								// Update `1st_effect_idx` on all subsequent abilities
								PATCH_WITH_SCOPE BEGIN
									FOR ("i" = 1 ; "%i%" < SHORT_AT 0x68 - "%ab_ind%" ; "i" += 1) BEGIN
										WRITE_SHORT ("%ab_off%" + "%i%" * "%ab_size%" + 0x20) (THIS + 1)
									END
								END
							END
						END
					END
				END
			END
		BUT_ONLY_IF_IT_CHANGES
	END
END

DavidW · July 26, 2022

15 hours ago, Luke said:

as far as subtracting is concerned, you showed us a handy way to achieve it (the 999 trick)

Are you suggesting I should put it in the document? I could do, but it's already 60 pages long and that's with a fairly active attempt to stay focused on the essentials.

15 hours ago, Luke said:

As far as adding is concerned, it might be convenient to use WeiDU’s low-level functionality instead of WeiDU's built-in functions (mainly for efficiency – the following piece of code should take less than 5 seconds to patch 3083 files (unmodded IWD:EE)...)

I don't think it's going to be *convenient* to do so. It might occasionally be *necessary*, either if there are no bespoke functions or (your case) if you need to patch thousands of files. But those are rare. Even the example you give is borderline - it takes 2.7 seconds (on BG2EE) on my computer, and I can do it in 8 seconds using native WEIDU CLONE_EFFECT and under 4 using SCS's version of CLONE_EFFECT. (As with the 999 trick, you want to do it intelligently: do a first pass using GET_OFFSET_ARRAY and only clone if you find a match.) Unless you're doing a lot of bulk edits like that (and very few mods are in the business of multiply editing thousands of files) those times are short enough that most people are going to be better off using functions. (Not that there's anything wrong with using the low-level functionality, but it does get fiddly quickly - especially if you're patching something more complicated than a SPL or ITM file.)

EDIT:

If you really do want to optimize for speed, I would lose all those PATCH_WITH_SCOPEs - a quick test gets nearly a 50% speedup without them. You need to do PATCH_CLEAR_ARRAY fx_data each time you loop through a new ability, and you can leave the outermost WITH_SCOPE in place if you want to insulate the rest of your code from this bit (or wrap it in a function, which is what I usually do).

Luke · July 26, 2022

7 hours ago, DavidW said:

Are you suggesting I should put it in the document? I could do, but it's already 60 pages long and that's with a fairly active attempt to stay focused on the essentials.

Yes, it might be useful to know this trick...

7 hours ago, DavidW said:

I don't think it's going to be *convenient* to do so. It might occasionally be *necessary*, either if there are no bespoke functions or (your case) if you need to patch thousands of files...

Yes, WeiDU’s low-level functionality should only be used when COPY_EXISTING_REGEXP a large amount of files... If you're dealing with just a bunch of files, then you'd be better off using WeiDU's built-in functions (which are certainly easier to read...)

7 hours ago, DavidW said:

... and I can do it in 8 seconds using native WEIDU CLONE_EFFECT and under 4 using SCS's version of CLONE_EFFECT...

Guess you could also opt for WeiDU ADD_SPELL|ITEM_EFFECT (it takes roughly 9 seconds on my computer...)

Spoiler

ACTION_TIME "clone_effect_ex" BEGIN
	WITH_SCOPE BEGIN
		COPY_EXISTING_REGEXP "^.+\.\(spl\|itm\)$" "override"
			PATCH_WITH_SCOPE BEGIN
				PATCH_MATCH "%DEST_EXT%" WITH
					"SPL" BEGIN
						GET_OFFSET_ARRAY "ab_array" SPL_V10_HEADERS
					END
					"ITM" BEGIN
						GET_OFFSET_ARRAY "ab_array" ITM_V10_HEADERS
					END
					DEFAULT
				END
				PHP_EACH "ab_array" AS "ab_ind" => "ab_off" BEGIN
					PATCH_WITH_SCOPE BEGIN
						GET_OFFSET_ARRAY2 "fx_array" "%ab_off%" SPL_V10_HEAD_EFFECTS
						PHP_EACH "fx_array" AS "fx_ind" => "fx_off" BEGIN
							PATCH_IF (LONG_AT ("%fx_off%" + 0x24) BAND (BIT0 BOR BIT1 BOR BIT2 BOR BIT3 BOR BIT4)) BEGIN
								// Save current values
								READ_SHORT ("%fx_off%" + 0x0) "fx_opcode"
								READ_BYTE ("%fx_off%" + 0x2) "fx_target"
								READ_BYTE ("%fx_off%" + 0x3) "fx_power"
								READ_SLONG ("%fx_off%" + 0x4) "fx_parameter1"
								READ_SLONG ("%fx_off%" + 0x8) "fx_parameter2"
								READ_BYTE ("%fx_off%" + 0xC) "fx_timing"
								READ_BYTE ("%fx_off%" + 0xD) "fx_resist_dispel"
								READ_LONG ("%fx_off%" + 0xE) "fx_duration"
								READ_BYTE ("%fx_off%" + 0x12) "fx_probability1"
								READ_BYTE ("%fx_off%" + 0x13) "fx_probability2"
								READ_ASCII ("%fx_off%" + 0x14) "fx_resource"
								READ_LONG ("%fx_off%" + 0x1C) "fx_max_level"
								READ_LONG ("%fx_off%" + 0x20) "fx_min_level"
								READ_LONG ("%fx_off%" + 0x24) "fx_savetype"
								READ_SLONG ("%fx_off%" + 0x28) "fx_savebonus"
								READ_SLONG ("%fx_off%" + 0x2C) "fx_special"
								TEXT_SPRINT $"fx_data"("%fx_opcode%" "%fx_target%" "%fx_power%" "%fx_parameter1%" "%fx_parameter2%" "%fx_timing%" "%fx_resist_dispel%" "%fx_duration%" "%fx_probability1%" "%fx_probability2%" "%fx_resource%" "%fx_max_level%" "%fx_min_level%" "%fx_savetype%" "%fx_savebonus%" "%fx_special%" "%fx_ind%") "irrelevant"
							END
						END
						// Add it
						PATCH_WITH_SCOPE BEGIN
							SET "fx_count" = 0
							PATCH_MATCH "%DEST_EXT%" WITH
								"itm" BEGIN
									PHP_EACH "fx_data" AS "fx_attributes" => "" BEGIN
										LPF "ADD_ITEM_EFFECT"
										INT_VAR
											"type" = 99 // all types
											"header" = "%ab_ind%" + 1 // count starts from 1 instead of 0!!!
											"opcode" = "%fx_attributes_0%"
											"target" = "%fx_attributes_1%"
											"power" = "%fx_attributes_2%"
											"parameter1" = "%fx_attributes_3%"
											"parameter2" = "%fx_attributes_4%"
											"timing" = "%fx_attributes_5%"
											"resist_dispel" = "%fx_attributes_6%"
											"duration" = "%fx_attributes_7%"
											"probability1" = "%fx_attributes_8%"
											"probability2" = "%fx_attributes_9%"
											"dicenumber" = "%fx_attributes_11%"
											"dicesize" = "%fx_attributes_12%"
											"savingthrow" = "%fx_attributes_13%"
											"savebonus" = "%fx_attributes_14%" - 2
											"special" = "%fx_attributes_15%"
											"insert_point" = "%fx_attributes_16%" + 1 + "%fx_count%" // insert below
										STR_VAR
											"resource" = "%fx_attributes_10%"
										END
										SET "fx_count" += 1
									END
								END
								"spl" BEGIN
									PHP_EACH "fx_data" AS "fx_attributes" => "" BEGIN
										LPF "ADD_SPELL_EFFECT"
										INT_VAR
											"header" = "%ab_ind%" + 1 // count starts from 1 instead of 0!!!
											"opcode" = "%fx_attributes_0%"
											"target" = "%fx_attributes_1%"
											"power" = "%fx_attributes_2%"
											"parameter1" = "%fx_attributes_3%"
											"parameter2" = "%fx_attributes_4%"
											"timing" = "%fx_attributes_5%"
											"resist_dispel" = "%fx_attributes_6%"
											"duration" = "%fx_attributes_7%"
											"probability1" = "%fx_attributes_8%"
											"probability2" = "%fx_attributes_9%"
											"dicenumber" = "%fx_attributes_11%"
											"dicesize" = "%fx_attributes_12%"
											"savingthrow" = "%fx_attributes_13%"
											"savebonus" = "%fx_attributes_14%" - 2
											"special" = "%fx_attributes_15%"
											"insert_point" = "%fx_attributes_16%" + 1 + "%fx_count%" // insert below
										STR_VAR
											"resource" = "%fx_attributes_10%"
										END
										SET "fx_count" += 1
									END
								END
								DEFAULT
									PATCH_FAIL "File not supported (~%DEST_FILE%~)"
							END
						END
					END
				END
			END
		BUT_ONLY_IF_IT_CHANGES
	END
END

7 hours ago, DavidW said:

If you really do want to optimize for speed, I would lose all those PATCH_WITH_SCOPEs - a quick test gets nearly a 50% speedup without them. You need to do PATCH_CLEAR_ARRAY fx_data each time you loop through a new ability...

Are you sure about this...?

Asking because it takes roughly 1 second more than before for me (as far WeiDU Timings are concerned, the most relevant difference between having all those PATCH_WITH_SCOPEs and not having them is `process_patch2` => 2.199 with PATCH_WITH_SCOPEs vs. 3.057 without PATCH_WITH_SCOPEs...)

Edited July 26, 2022 by Luke

Sign In

A Course in WEIDU

Recommended Posts

CamDawg

Link to comment

DavidW

Link to comment

AL|EN

Link to comment

AL|EN

Link to comment

Luke

Link to comment

DavidW

Link to comment

Luke

Link to comment

DavidW

Link to comment

Salk

Link to comment

Magus

Link to comment

mickabouille

Link to comment

Awachi

Link to comment

Luke

Link to comment

DavidW

Link to comment

Luke

Link to comment

Join the conversation

Website

Forums

My Activity Streams

Downloads

Gallery