editing item description strings

[subtledoctor]

Okay, this might well be a very, very simple question that I should know the answer to. But I can't quite figure it out from the Weidu docs. Can I in Weidu, and if so is there a best practice for, doing a massive Find/Replace across every string in the game?

 

Let's say, for example, I want to find every instance of " +1" (that's space-plus-one) and change it to " + 1" (space-plus-space-one). And I want to do it like a regexp search, so that it affects any strings like spell and item descriptions that may be added by other mods.

 

Possible?

 

Method?

 

thx peeps

[Jarno Mikkola]

Possible ? Yes.
Method, well, you could try something that uses the EVALUATE_BUFFER_SPECIAL string ... but it can fail if the game has a lots of mod content, or has multiple repeated mod un/re-installs as the normal dialog.tlk can grow up a lots(it's normal size is 8.2 MB's, but if it ever grows above the 16MB's, the mod will definitely fail, as the weidu.exe can't patch files that large).

[subtledoctor]

Hm. Well, poop. So, if anyone feels like brainstorming, let's spitball here. What I ultimately want to do, is to eliminate every "+1" and "+2" etc. from every item description in the game. Like, why did they call it "Blade of Roses +3?" Why not just "Blade of Roses" and when you read the description you can see that it has +3 thac0 and +3 damage? Much less immersion-breaking that way.

 

I could just repeat a ton of STRING_SET commands, item-by-tem... a lot of grunt work on my end but it would get the job done for an unmodded game. However, that wouldn't work with any items that had already been subject to a COPY_EXISTING with a SAY NAME1. Maybe I could go through every item added by any mod with an ACTION_IF FILE_EXISTS_IN_GAME but... my god, how many items are we talking about here?

 

I suppose you could script some kind of loop that goes string-by-string and looks for the offending phrases, that might be more digestible for Weidu... But it would have to repeat what, ~30,000 times? That might take a while to execute...

 

If anyone has any ideas, please let me know!

[CamDawg]

The good news (for me at least) is that it looks like you're angling to undo something I argued against at length in BG(II)?EE, which I'm happy to see. I hate, hate, hate seeing a +x on a unique item name, as it implies it's one in a series, e.g. Kondar +1 implies that somewhere out there is a Kondar +2. I had planned to add a tweak for it and simply never got to it.

 

BG2 Tweaks does have multiple places where it adjusts descriptions on the fly, but the code is not for the faint of heart. Here's a fairly simple case from Allow Arcane Casting in Armor--at this point all of the changes have already been made to the armor, and we're just adjusting the description:

 

        // adjust descriptions
        FOR (index = 0x50 ; index < 0x55 ; index = index + 0x04) BEGIN
          READ_LONG "%index%" "valid"
          PATCH_IF ("%valid%" < 2147483646) AND ("%valid%" >= 0) BEGIN
            READ_STRREF "%index%" "description"
            INNER_PATCH ~%description%~ BEGIN
              LAUNCH_PATCH_MACRO ~arcane_descripts~
              SET char = 0 // different than the check
              FOR (i = 0; char != 0xfafa; i += 1) BEGIN
                READ_BYTE i char ELSE 0xfafa // if out of bounds, read my custom EOF signal/value
              END
              READ_ASCII 0 description (i - 1)
            END
            SAY_EVALUATED "%index%" ~%description%~
          END

And relevant tpa and tra:

DEFINE_PATCH_MACRO ~arcane_descripts~ BEGIN

SPRINT arcane_weight @212001
SPRINT arcane_miscast @212002
REPLACE_TEXTUALLY ~%arcane_weight%~
~\1
%arcane_miscast% +%patch_miscast%%~

END

@212001 = ~\(Weight:[ %tab%]*[0-9]+\)~
@212002 = ~Miscast Arcane Magic:~

So, what are we looking at here?

 

FOR (index = 0x50 ; index < 0x55 ; index = index + 0x04) BEGIN
  READ_LONG "%index%" "valid"
  PATCH_IF ("%valid%" < 2147483646) AND ("%valid%" >= 0) BEGIN
    READ_STRREF "%index%" "description"

We set up a brief loop that reads in the strref from the unidentified and identifed fields of an item (0x50 and 0x54). We make sure the strref isn't insane (e.g. a negative value or way beyond the end of the string table), and if it's not we read in the actual contents of the string insto the variable 'descripiton'. Now, the fun part where we invoke the macro--in this case I've gone ahead and made the string subs to make it a little more straightforward:

 

DEFINE_PATCH_MACRO ~arcane_descripts~ BEGIN

  SPRINT arcane_weight ~\(Weight:[ %tab%]*[0-9]+\)~
  SPRINT arcane_miscast ~Miscast Arcane Magic:~
  REPLACE_TEXTUALLY ~%arcane_weight%~
    ~\1
%arcane_miscast% +%patch_miscast%%~

END

So what we're doing here is looking for the 'Weight: X' line in the description, preserving it, and adding the 'Miscast Arcane Magic: +X%' line below it (the patch_miscast variable was set earlier). Following this is some custom code from the bigg which I've been assured is a good idea:

 

SET char = 0 // different than the check
FOR (i = 0; char != 0xfafa; i += 1) BEGIN
  READ_BYTE i char ELSE 0xfafa // if out of bounds, read my custom EOF signal/value
END
READ_ASCII 0 description (i - 1)

followed by the actual writing of the string back onto the item:

 

SAY_EVALUATED "%index%" ~%description%~

Since SAY does not evaluate variables in strings, you need SAY_EVALUATED.

 

Now, if you're looking for a global search and replace across everything, in some ways that's a little easier. First you can dynamically find the end of the tlk with something like

 

COPY_EXISTING ~sw1h01.itm~ ~override/temp.itm~
  SAY 0x08 ~fkgljdslgjsdlgjflgjlsdgjl~ // some junk string that's not going to be in the tlk anywhere
  READ_LONG 0x08 end // since it should be a new string, it should be last in the tlk table

Then it's a mater of just looping and testing:

 

OUTER_FOR (index = 0 ; index < end ; ++index) BEGIN

  ACTION_GET_STRREF index string

  // junk to change string

  STRING_SET_EVALUATED index "%string%"

END

Like I said, it's not for the faint of heart.

[Mike1072]

If you're just worried about the item names, looping through all .itm files to read and update their identified names is easiest. It's exactly the same as patching descriptions, you just read from a different offset.

In IR, I used a modified version of the description patch CamDawg posted above. The only real difference is using the command INNER_PATCH_SAVE instead of INNER_PATCH to make things simpler.

This:

INNER_PATCH ~%description%~ BEGIN
  LAUNCH_PATCH_MACRO ~arcane_descripts~
  SET char = 0 // different than the check
  FOR (i = 0; char != 0xfafa; i += 1) BEGIN
    READ_BYTE i char ELSE 0xfafa // if out of bounds, read my custom EOF signal/value
  END
  READ_ASCII 0 description (i - 1)
END
SAY_EVALUATED "%index%" ~%description%~

Becomes this:

INNER_PATCH_SAVE new_desc ~%description%~ BEGIN
  LAUNCH_PATCH_MACRO ~arcane_descripts~
END
SAY_EVALUATED "%index%" ~%new_desc%~

(Apparently when you wrote this back in 2005, Cam, INNER_PATCH_SAVE hadn't been implemented yet.)

[CamDawg]

Oh, you young'uns, you don't know how we had to hike six miles to the tp2 in snow, uphill both ways. I'm still bitter I missed my beer for v175.

 

Seriously, though, I don't even remember being the impetus for this.

[subtledoctor]

Wow, thanks guys. I'm going to have to digest this a bit, and maybe work on it this weekend.

 

CamDawg, I will hand it to you guys, EE is a lot better than the OG for this kind of stuff. It moves in the right direction. Of course the various changes do complicate things for modders trying to target all game variants, which is why I have stuff like this in my .tp2:

 

ACTION_IF FILE_EXISTS_IN_GAME ~~ THEN BEGIN
...
END

ACTION_IF GAME_IS ~bgee bg2ee~ THEN BEGIN
...
END

ACTION_IF GAME_IS ~tutu bgt soa tob~ THEN BEGIN
...
END

ACTION_IF GAME_IS ~bgt tob bg2ee~ THEN BEGIN
...
END

 

 

Of course the simple text changes aren't so simple either; because there's tons of places where I need to preserve "+1" phrases. So I have to go in and change all of these from "+1" to "+ 1" so as not to be caught in the net:

 

 +1 to
 +2 to
 +3 to
 +4 to
 +5 to
 +1 bonus
 +2 bonus
 +3 bonus
 +4 bonus
 +5 bonus
of +1
of +2
of +3
of +4
of +5
: +1
: +2
: +3
: +4
: +5
 +1 THAC0
 +2 THAC0
 +3 THAC0
 +4 THAC0
 +5 THAC0
 +1 to
 +2 to
 +3 to
 +4 to
 +5 to
 +1 vs.
 +2 vs.
 +3 vs.
 +4 vs.
 +5 vs.
 +1 cold
 +1 electrical
 +1 fire
 +1 acid
 +1 magic
 +1 slashing
 +1 piercing
 +1 crushing
 +5%
 +10
 +15
 +20
 +25
 +50
 +100
 +1 Str
 +1 Dex
 +1 Con
 +1 Int
 +1 Wis
 +1 Cha
, +1
, +2
, +3
, +4
, +5
 +1d
 +2d
 +3d
 +4d
 +5d
 +1 round
THAC0 +1
THAC0 +2
THAC0 +3
THAC0 +4
THAC0 +5
AC +1
AC +2
AC +3
AC +4
AC +5

 

 

... and then finally go through and replace every "+1" with " " Which, by my count, means I have to run that macro ~75 times to preserve those strings, then run it 5 times to change the item descriptions, then run it 5 more times to reset the preserved strings. I'm really excited to find out how long this component takes to install.

 

EDIT: Ah I see, if I only worry about identified item names, I don't have to worry about preserving anything. Of course then the description will still say silly stuff like "Short Sword of Backstabbing +3" followed by the rest of the description. I hate that stuff: it's in-game lore, what does "+3" mean to a character in-game? It's weird. The technical information after the flavor text has all the info a player needs, and because I'm standardizing enchantment levels, the player doesn't need that "+3" at all.

 

Ah crap. I'm standardizing enchantment levels. So if I leave the "+1"s in the identified descriptions, it might actually deceive the player. Sigh. Not to mention there are spells and other places that talk about "sword +2" that should be changed as well. So, I guess I really do need to do broad changes across all strings.

[subtledoctor]

 

DEFINE_PATCH_MACRO ~arcane_descripts~ BEGIN

  SPRINT arcane_weight ~\(Weight:[ %tab%]*[0-9]+\)~
  SPRINT arcane_miscast ~Miscast Arcane Magic:~
  REPLACE_TEXTUALLY ~%arcane_weight%~
    ~\1
%arcane_miscast% +%patch_miscast%%~

END

So what we're doing here is looking for the 'Weight: X' line in the description, preserving it, and adding the 'Miscast Arcane Magic: +X%' line below it (the patch_miscast variable was set earlier).

Okay, so, if I don't want to preserve anything, but just replace " +1" with " ", do I need those 'sprint' lines? Indeed, do I need the macro at all? Could I just write:

COPY_EXISTING_REGEXP GLOB ~.*\.itm~ ~override~
    READ_LONG 0x54 "valid"
        PATCH_IF ("%valid%" < 2147483646) AND ("%valid%" >= 0) BEGIN
            READ_STRREF 0x0c "description"
            INNER_PATCH_SAVE new_desc ~%description%~ BEGIN
                REPLACE_TEXTUALLY ~ +1~ ~ ~
            END
            SAY_EVALUATED 0x54 ~%new_desc%~
        END
        BUT_ONLY

...and then repeat for +2, +3, etc.? (This is for a watered-down version where I'm only changing identified descriptions.)

 

Except that doesn't do anything. It installs without errors, but doesn't change anything.

[Mike1072]

REPLACE_TEXTUALLY defaults to doing a regular expression replacement, so ~ +1~ means it is looking for one or more spaces followed by a 1, when you probably wanted it to look for a space followed by a plus sign followed by a 1.

 

You'll want to take advantage of regular expressions, since it avoids the need to do separate replacements. You could use something like ~ \+[1-6]~ which will look for a space followed by a plus sign followed by the number 1, 2, 3, 4, 5, or 6. This will catch a lot of things in the description that you don't want to remove, but it's a starting point.

 

With IR, we needed to do many different replacements and we wanted to reduce code duplication, so we modified Cam's code such that we have 1 macro that actually READs and WRITEs the descriptions and many other macros that perform specific substitutions.

 

It looks something like this now. The update_item_descriptions macro doesn't need to know what text is actually being replaced. It takes the name of the macro that performs the substitution as a parameter (text_update).

DEFINE_PATCH_MACRO ~update_item_descriptions~ BEGIN
  FOR (index = 0x54 ; index >= 0x50 ; index -= 4) BEGIN // loop through descriptions
    READ_LONG "%index%" "valid"
    PATCH_IF ("%valid%" < 2147483646) AND ("%valid%" >= 0) BEGIN // verify description is valid
      READ_STRREF "%index%" "description"
      INNER_PATCH_SAVE new_desc ~%description%~ BEGIN
        LAUNCH_PATCH_MACRO ~%text_update%~
      END
      SAY_EVALUATED "%index%" ~%new_desc%~
    END
  END
END

This is an example of a macro that performs a substitution. It doesn't need to know anything about offsets or reading/writing values, just replacing. Note that if you need to perform multiple replacements, you should put all those REPLACE_TEXTUALLY statements in here rather than in separate macros, so that you minimize the number of strings added to dialog.tlk.

DEFINE_PATCH_MACRO ~remove_enchantment_level~ BEGIN
  REPLACE_TEXTUALLY ~ \+[1-6]~ ~~
END

With that in place, you would use it something like this.

COPY_EXISTING_REGEXP GLOB ~^.+\.itm$~ ~override~
  PATCH_IF (SOURCE_SIZE > 0x71) BEGIN
    SPRINT text_update remove_enchantment_level
    LPM update_item_descriptions
  END
  BUT_ONLY

For your purposes, you may want to use a modified version of update_item_descriptions that edits the identified name (0x0c) and identified description (0x54) fields rather than the unidentified description (0x50) and identified description (0x54) fields.

[subtledoctor]

Ah. Okay. That looks reasonably easy to add (i.e. paste) into my tp2.

 

But, let's assume for the moment that I'm I'm only going to patch a single offset (0x0c, because dealing with false positives in 0x54 is not, for the moment, worth the effort), with a single substitution rule (~ \+[1-6]~ ~~)... is the macro *needed* in that case? It doesn't hurt to include it of course, so this is really an academic question. I just feel like it's broadly good practice to keep the code as simple as possible, so that when I or someone else looks at it a year from now with no clue/memory what it's doing, it'll be easier to understand.

[CamDawg]

Macros are never actually needed; it's just an easy way to call code you need more than once. This is very good for maintenance reasons--if you're doing the same operations over and over, you're better off writing a macro and calling it as needed, rather than trying to maintain X copies of the same code in a tp2.

 

Or, you can go super-meta like DavidW and write macros that, in turn, write new macros that call more macros. I think when David adds another level of macros, his code will turn into gray goo and eat us all.

[subtledoctor]

Alright well cutting and pasting Mike's stuff was giving me parse errors. Since for the moment I'm only worrying about item *names* and I want to get the code working before worrying about how to organize it most efficiently (i.e. macro vs. simple command), I distilled it down to this:

//ITEM DESCRIPTIONS PATCH____________________
//
COPY_EXISTING_REGEXP GLOB ~^.+\.itm$~ ~override~
	PATCH_IF (SOURCE_SIZE > 0x71) BEGIN
		READ_LONG 0X0c "valid"
		PATCH_IF (%valid% >= 0) BEGIN
			READ_STRREF 0x0c "name"
			INNER_PATCH_SAVE new_name ~%name%~ BEGIN
				REPLACE_TEXTUALLY ~ \+[1-6]~ ~~
			END
			SAY_EVALUATED 0x0c ~%new_name%~
		END
	END
//______________________________________

This works beautifully to remove any +1s from items' identified names. (A few will need special handling, like "Bone Club +2, +3 vs. Undead" which is now "Bone Club, vs. Undead")

 

But looking in NI, I see some undesired changes to other items. Basically items that are designed to be known by their unidentified named - some simly have "-1" in the NAME2 field, but others have something else, which shows up in NI as "no such string" (or something like that). Examples include the gem bag (I think bag2i.itm) and a couple axes, I think throwing axes (ax1h16.itm and ax1h17.itm)

 

The above patch changes those NAME2 fields to something like "invalid reference (4578361)". The SAY_EVALUATED is running on every item, writing their original name back into the NAME2 field if they don't get altered by the text substitution. But for items without a valid string in that field, this script is turning weird into weirder.

 

Now this may not matter since such items are only known by their unidentified names. But, I don't want to change things that don't need changing.

 

So is there a way to set a condition for patching that the NAME2 string actually includes ~ \+[1-6]~, and only in those cases apply the text substitution and the SAY_EVALUATED command? Something like, just before the INNER_PATCH_SAVE line,

PATCH_IF ~%name%~ contains ~ \+[1-6]~ BEGIN

 

In other words: how might I implement that "contains" condition?

[subtledoctor]

Hmm, maybe I needed to google harder/faster/stronger. Looks I could maybe use

PATCH_IF ~%name%~ STRING_CONTAINS_REGEXP ~\+[1-6]~ =0 BEGIN
        INNER_PATCH_SAVE etc.

[Mike1072]

Alright well cutting and pasting Mike's stuff was giving me parse errors.

It shouldn't. I tested it.

 

...

This works beautifully to remove any +1s from items' identified names. (A few will need special handling, like "Bone Club +2, +3 vs. Undead" which is now "Bone Club, vs. Undead")

 

But looking in NI, I see some undesired changes to other items. Basically items that are designed to be known by their unidentified named - some simly have "-1" in the NAME2 field, but others have something else, which shows up in NI as "no such string" (or something like that). Examples include the gem bag (I think bag2i.itm) and a couple axes, I think throwing axes (ax1h16.itm and ax1h17.itm)

 

The above patch changes those NAME2 fields to something like "invalid reference (4578361)". The SAY_EVALUATED is running on every item, writing their original name back into the NAME2 field if they don't get altered by the text substitution. But for items without a valid string in that field, this script is turning weird into weirder.

You really should not have that many items with invalid strings (other than -1). The only ones you might have like that should be from items players can never get. These are the ones like that in my Fixpacked install:

 

 

 

B1-10.ITM
B1-12.ITM
B1-2.ITM
B1-20.ITM
B1-3.ITM
B1-4.ITM
B1-8.ITM
CARRIO.ITM
ETTERC.ITM
GBASIL.ITM
GHOULC.ITM
GISPID.ITM
HUSPID.ITM
P1-10.ITM
P1-12.ITM
P1-2.ITM
P1-20.ITM
P1-3.ITM
P1-6.ITM
P2-16.ITM
PHSPID.ITM
RNDTRE01.ITM
RNDTRE02.ITM
RNDTRE03.ITM
RNDTRE04.ITM
RNDTRE05.ITM
SPIDWR.ITM
SWSPID.ITM

 

 

However, you still have a good point. There's no need to add unnecessary strings to dialog.tlk. I added an extra condition to my macro to ignore these invalid references:

 

DEFINE_PATCH_MACRO ~update_item_descriptions~ BEGIN
  FOR (index = 0x54 ; index >= 0x50 ; index -= 4) BEGIN // loop through descriptions
    READ_LONG "%index%" "valid"
    PATCH_IF ("%valid%" < 2147483646) AND ("%valid%" >= 0) BEGIN // verify description is valid
      READ_STRREF "%index%" "description"
      PATCH_IF (~%description%~ STRING_COMPARE_REGEXP ~<Invalid Strref [0-9]+>~ == 0) BEGIN // more validation
        INNER_PATCH_SAVE new_desc ~%description%~ BEGIN
          LAUNCH_PATCH_MACRO ~%text_update%~
        END
        SAY_EVALUATED "%index%" ~%new_desc%~
      END
    END
  END
END

You don't need to worry about valid strings that don't match your criteria. When instructed to write a string, WeiDU will look it up in dialog.tlk first. If it finds an existing reference for the string, it will use that. Otherwise, it will add the string and give it a new reference. (So, descriptions that are not changed by your REPLACE_TEXTUALLY statement will have the same strref written to the item as they started with.)

[subtledoctor]

Sorry, I shouldn't have said cut and pasted; I changed it because your macro loops through bits 0x50 and 0x54, whereas mine only dealt with 0x0c. I never quite got the first line to where Weidu liked it, and I kept fiddling with it until I realized I could simply omit that line and jump right to READ_LONG 0x0c, and that worked fine. So the parse error wasn't cause by your code, but rather my inability to adapt it to my circumstances.

 

Anyway thanks. That condition will work, looks like. Here's what I ended up with, it's working nicely:

//PATCH ITEM NAMES__________________________
//
COPY_EXISTING_REGEXP GLOB ~^.+\.itm$~ ~override~
   PATCH_IF (SOURCE_SIZE > 0x71) BEGIN
      READ_LONG 0x0c "valid"
      PATCH_IF (%valid% >= 0) BEGIN
         READ_STRREF 0x0c "name"
         PATCH_IF (~%name%~ STRING_CONTAINS_REGEXP ~\+[1-6]~) = 0 BEGIN
            INNER_PATCH_SAVE new_name ~%name%~ BEGIN
               REPLACE_TEXTUALLY ~ \+[1-6]~ ~~
            END
            SAY_EVALUATED 0x0c ~%new_name%~
         END
      END
   END

Patching descriptions is going to be a whole other can of worms. The effort just isn't worth it. Realistically, the identified names are seen by the player much more often, so I'll compromise and leave the descriptions slightly immersion-breaking - for now.