eet Lurker's EET install: eet-modpack_2023-09-17_v2

[Lurker]

Is there an easy way to fix StringRefs in saved games with Near Infinity, or another tool?

The changes between my installations boil down to a static offset of -656 that needs to be applied to mismatching strings in my ongoing game, at least as far as I currently understand this.

EDIT: When I compare the WeiDU.logs of my old setup against the current one, all StringRefs up until the first change should match, I think. So anything after this would need to be adjusted, if already present in the saved game. Am I screwed...?

EDIT: This boils down to

• Worldmap data in BALDUR.SAV (can fix manually, but is tedious)
• individual characters in BALDUR.gam (can fix manually, but is VERY tedious)
• names of non-joinable NPCs - where are they stored?
• names of Shopkeepers and Shops - where are they stored?
• what am I missing?

Edited October 29, 2023 by Lurker

[jmerry]

Only the stuff in the actual save would need to be changed.

- Creatures. Either in the .GAM file for joinables and other global character, or in the various areas in the .SAV file. Creature names are a string reference field in the .CRE file.

- Shops. Shopkeepers are creatures as above, and the name of the shop itself is a string reference field in the .STO file. Stores that you've opened up are also in the .SAV file.

But really, it's a tedious mess no matter what. Trying to change a mod install in the middle of a run is just not recommended. Better to start over.

[Lurker]

4 hours ago, jmerry said:

Better to start over.

What if I just tried to reinstall again and pad dialog.tlk with 656 dummy entries at the right stage...?

 

[Lurker]

I'm stubbornly refusing to give up on my current game. Experiments with weidu --strapp look promising so far, but some mods make my head scratch, in that their strings in dialog.tlk aren't deterministically placed, as far as I currently understand it. The mods that are causing me a bit of grief are bst (some variance) and blackhearts_eet (huge variance).

If I repeated the same installation process with the same number and order of mods, shouldn't the result be... identical? Is this a quirk of WeiDU?
 

[subtledoctor]

If some mod has an update (you are reinstalling to get updates, no?) and adds or subtracts a single string, then all strings will be wrong thereafter. The only way to get the same .tlk is to install precisely identical versions of a precisely  identical mod list, in the precisely identical order. 

For me, the only adjustments I make once a game is underway are by creating hotfix mods, or by manual adjustments in NI. (I just started an IWD run with two MnG revised bards, and found out that the 5E Casting mod had a compatibility error. I easily updated the mod, but fixing things in the game-in-progress was a minor nightmare.)

Are you not already into ToB? I forget what your thread said. lf I was that far along I’m not sure I would put the effort in…

[Lurker]

2 minutes ago, subtledoctor said:

Are you not already into ToB?

My current game was on pause, because I wanted to do a ToB run, because I was curious about Longer Road and also wanted to check/prevent that my full run would end in a messy nightmare comparable to the one before that, due to Transitions. The run went fine, but the final battle was completely broken, so I decided that I want to ditch anything from Lauriel.

I'm in Chapter 5. Too much time invested to abandon the game, but enough content untouched, so that the save should be salvageable. Since I removed more content than I added, I'm experimenting with "padding" dialog.tlk at convenient breakpoints in a reinstall.

[Lurker]

Hmm, I don't get it.

• I remove a mod from my install, that let's say had 500 entries in dialog.tlk
• so I reinstall, and in place of the removed mod, I add 500 dummy entries to dialog.tlk
• I've a chunk of mods installed after this, and they install their strings in the same order as in the previous installation
  • this includes heavy stuff like DSotSC and NTotSC
    • almost... a single string (@90600) from DSotSC messes up a journal entry
• then there comes Balduran's Sea Tower, that just doesn't care to place it's string where they should be, mixing things up with a deviation of around +-14
• right after that comes Shadows over Soubar, that again fits right in
• later comes blackhearts_eet, with some strings all over the place

I obviously haven't checked all string references manually, so I might miss something here. Are new strings always appended to dialog.tlk, or are there exceptions? Can there be "holes" in dialog.tlk, or things like "I want position 215000, and if that's already in use, I take the next free spot"? (Why?)

Edited October 31, 2023 by Lurker

[Lurker]

Well, it's not perfect, but compared to before, there are only very few things that need manual fixing with NI. Major improvement, and I learned... err... something.

The "padding" can be done directly in the wmodinstall payload (additional lines just for better context; relevant are the for loops):

 

modinstall c#endlessbg1 "0 1 2 5 6 7 8 10 11 12 13 16" 0
#modinstall transitions "0 10 20 30 200 40 50 60 70 71 72 73 80 140" 0
#^^^^^^^^^^ mod breaks ToB in EET; author retired; mod is unsupported
modinstall c#endlessbg1 "3 4 14 15" 0
#^^^^^^^^^ components that would normally break transitions
modinstall c#sodtweaks "3 4 5 11 20 21" 0
modinstall c#sodboabri "0 1 2 3" 0
modinstall alternatives "0" 0
modinstall imoen_forever "0 1 2 3 9 10 11 12 13 14 20 25" 0
#^^^^^^^^^ readme describes obsolete but not new components; that is very un-jastey-like
for CNT in `seq 1 394`; do weidu --strapp PADDING_FOR__transitions__AFTER__imoen_forever__$CNT; done
modinstall jaheirarecast "1 2" 0

 

modinstall banterpack "0 2" 0
#modinstall themed_tweaks "60 70 80 90 100 140 180" 0
#^^^^^^^^^^ author retired; mod is unsupported; better safe than sorry
for CNT in `seq 1 262`; do weidu --strapp PADDING_FOR__themed_tweaks__AFTER__banterpack__$CNT; done
modinstall bgqe "0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16" 0

 

modinstall sarevefffix "0" 0
modinstall tidings "0" 1
#modinstall longerroad "0 1" 0
#^^^^^^^^^^ possibly completely breaks ToB endgame battle; backtick text encoding problem; Irenicus' Project Image lacks spells?
for CNT in `seq 1 1804`; do weidu --strapp PADDING_FOR__longerroad__BEFORE__totemiccernd__$CNT; done
modinstall totemiccernd "0" 0

 

I also moved down Dark Tidings a little bit, to more closely replace Longer Road's slot.

Individually calling weidu --strap over 2000 times isn't very efficient due to cumulative overhead, and takes some time. But it's the simplest solution I was able to come up with.

[subtledoctor]

3 hours ago, Lurker said:

wanted to check/prevent that my full run would end in a messy nightmare comparable to the one before that, due to Transitions. The run went fine, but the final battle was completely broken, so I decided that I want to ditch anything from Lauriel

Huh, well I've got Transitions installed in my game... wait, what final battle was broken - Irenicus? Or Amelissan?

[CamDawg]

Right, so the string situation has some subtleties that make this whole thing a windmill tilt. Some of this is information you already know, but I'll make this a bit of a primer for anyone interested in how this actually works.

Game text is stored in the tlk table. It's basically an index that says string #9501 has the contents of "Minsc". Minsc's creature file does not contain the text "Minsc" in the name field, but instead refers the string reference #9501. When the engine needs to show the name for the Minsc creature file, it looks up #9501 on the table and uses the text listed there: "Minsc".

When you install a WeiDU mod, there are all sorts of strings used by the mod--spell names, item descriptions, NPC dialogue, whatever. Many of these are new, but WeiDU also gives the ability to overwrite or modify an existing string. For example if you're changing something to the base long sword description, it's probably better to change the string instead of assigning a new string to the long sword,. This is because the base long sword description is used as the unidentified description for all magical long swords. Things like descriptions for existing kits are usually updated, since it's easier than making a new string and chasing references to it through a bunch of UI and rule tables.

So mods end up using a combination of new strings and updating existing strings. However, if my mod says I need a new string for something, WeiDU only adds the string if it's not already in the tlk table. In other words, if I tell the engine I need a new string of "Imoen" for an Imoen copy I need for a new cutscene, WeiDU doesn't add the string. It points out that we already have a perfectly serviceable, existing string containing "Imoen" and uses that instead. If I request a string that's not on the table then it adds it to the end of the tlk and uses the newly created reference.

Importantly, uninstalling a mod does not behave as you might expect. Strings which were changed during install are changed back to their pre-mod contents, but any new strings that were added are left, intact, in the tlk table. (Example: BG2EE ships with ~102k strings. My BG2EE copy for testing Tweaks currently has no mods installed, but is now up to ~110k tlk strings due to leftovers from mods that had been previously installed.). This is meant to try and preserve the tlk table for mods after it--e.g. imagine installing mods A, B, and then C, and then removing A. The references for B and C would now be completely different and utterly break save games. We try to discourage continuing on old saves after changing mods, but this is still meant as a measure to try and make bugs less likely for players that do it anyway.

All of this behavior leads to some subtle interactions between mods. Let's take an actual, real interaction between two mods you're using: The Calling and Balduran's Seatower, As you've previously noted, both mods add Black Pearls to the game. I'll make up some numbers to keep the math simple:

• Your tlk, fresh from a clean install, has 40000 strings. The Calling installs and requests, let's say, 100 new strings. One of those strings is the name of a new item, "Black Pearl". Let's say the installer assigns it to #40005 in the new block of strings it creates in the range of 40001-40100.
• You then install Balduran's Seatower, which also adds a Black Pearl. BST asks for 200 strings but WeiDU notes that "Black Pearl" is already in the tlk. So it assigns the BST Black Pearl a name string of 40005, and then adds 199 new strings to cover the rest of the requests. So BST now occupies the 40101-40299 range.
• You decide (correctly) that The Calling is run by a clown, and you don't want it any more. You start clean from a new install and its unmolested 40000-string tlk. In order to use your old saves, you decide to be clever and add 100 blank strings to cover The Calling's old string range. This will cause BST to start at 40101, like it did on your old install.
• Except it doesn't work. BST comes along, asking for 200 strings again, and getting the full 200 this time instead of 199, because 'Black Pearl' is not present and can't be recycled. Depending on where specifically "Black Pearl" gets slotted, you now have from 0-199 strings now one off from their previous values.

This is one of the easier ones to catch. When you consider the sheer number of small, semi-unique short lines of dialogue (especially replies)--"right away" or "I don't think so"--that could be lurking in dialogue-heavy mods, it's an intractable problem.

[Lurker]

Wow @CamDawg, thank you for taking the time to do this write-up. Stuff like that should really be collected in a read-only section in the forum, like perhaps Modding Q&A. These are valuable resources.

9 hours ago, CamDawg said:

However, if my mod says I need a new string for something, WeiDU only adds the string if it's not already in the tlk table.

Which doesn't mean that a string cannot have duplicates in dialog.tlk that were/are already present, only that WeiDU won't add new duplicates, correct? Example "Keelor", which I guess originates from BG1(*), at position 15625 and 15670, and gets again imported from BG1 during installation of EET to positions 215625 and 215670? This example raises the following questions:

• related to WeiDU:
  • Does WeiDU always use the first match when checking for existing strings in dialog.tlk as reference? Meaning if I want to add "Keelor", WeiDU references to 15625? Or are there boundaries depending on context, like EET never uses strings below a certain position? No, because WeiDU doesn't know what EET is.
• related to dialog.tlk and EET:
  • (*)There are also strings in close proximity that deal with BG2 content, and since I installed EET on top of BG2, the strings at the lowest positions are from BG2, yes? Meaning that there are leftovers from BG1 in a dialog.tlk that was repurposed for BG2 from the original game devs?
  • There are matches for "Keelor" at (215088, 215089, 215737, 215738, 215739) with no corresponding matches at (position - 200K), in the range that I'd expect to be EET's importing area. Are these present in the dialog.tlk of BG1, but not in the one from BG2, because the slots have been repurposed for BG2?
• (Not a question, just trying to be thorough:)
  • Starting at 30958, there are references to "Keelor the Dwarf"; these are from NTotSC in my installation

 

10 hours ago, CamDawg said:

Importantly, uninstalling a mod does not behave as you might expect.

Which is another reason why I don't rely on uninstalling - only in special cases and when close to the "onion's skin". Scripted reinstalls ftw.

10 hours ago, CamDawg said:

This is meant to try and preserve the tlk table for mods after it--e.g. imagine installing mods A, B, and then C, and then removing A. The references for B and C would now be completely different and utterly break save games. We try to discourage continuing on old saves after changing mods, but this is still meant as a measure to try and make bugs less likely for players that do it anyway.

Err... Does that mean I could have just copied over my dialog.tlk from my previous installation (v1) to my template for v2, and then just applied the slightly altered payload file?    (Need to check this out.)

10 hours ago, CamDawg said:

You decide (correctly) that The Calling is run by a clown

I would never do that, of course. Clowns are scary.

10 hours ago, CamDawg said:

In order to use your old saves, you decide to be clever and add 100 blank strings

Semi-clever. I didn't expect it to be perfect, but to ease the work of fixing references with NI. While powerful, the user interface of NI is rough.

I ran my old saves repeatedly against the new install, and in a mix of random and educated guesses, made notes about string mismatches, which I then compared with two different running instances of NI, each using the corresponding game. I was lucky (mostly related to mod install order) and discovered that I mostly could get away with fixed number for adjusting most strings. While this was akin to work, having to redo the same game over again would certainly feel so; perhaps I need to switch games when starting over feels more like work than fun. Will see if the effort was mostly a waste of time, if I discover more serious issues when actually trying to continue this playthrough.

Perhaps it pays off that I usually try to stick with a quest or complete an area, and hate branching off. The only issue I see with BST so far, for example, is the easily fixable entry on the worldmap, and since I haven't been to the actual area (only for testing the padded setup), the save game is still "untainted". DSotSC, NTotSC and Blackhearts could become problematic.

I could also provide a detailed diff between both WeiDU.log of v1 and v2, if someone is willing to help identifying the remaining discrepancies, if only to help me reducing that still present itching in my brain...

 

[Lurker]

11 hours ago, subtledoctor said:

Huh, well I've got Transitions installed in my game... wait, what final battle was broken - Irenicus? Or Amelissan?

Take a look at this topic and especially this comment. The final battle was working, every thing else, not so much. It's almost the complete opposite of what I describe here (ff).

Also, I did send you a PM yesterday, regarding something completely different.

Edited October 31, 2023 by Lurker

[CamDawg]

43 minutes ago, Lurker said:

Which doesn't mean that a string cannot have duplicates in dialog.tlk that were/are already present, only that WeiDU won't add new duplicates, correct? Example "Keelor", which I guess originates from BG1(*), at position 15625 and 15670, and gets again imported from BG1 during installation of EET to positions 215625 and 215670? This example raises the following questions:

• related to WeiDU:
  • Does WeiDU always use the first match when checking for existing strings in dialog.tlk as reference? Meaning if I want to add "Keelor", WeiDU references to 15625? Or are there boundaries depending on context, like EET never uses strings below a certain position? No, because WeiDU doesn't know what EET is.
• related to dialog.tlk and EET:
  • (*)There are also strings in close proximity that deal with BG2 content, and since I installed EET on top of BG2, the strings at the lowest positions are from BG2, yes? Meaning that there are leftovers from BG1 in a dialog.tlk that was repurposed for BG2 from the original game devs?
  • There are matches for "Keelor" at (215088, 215089, 215737, 215738, 215739) with no corresponding matches at (position - 200K), in the range that I'd expect to be EET's importing area. Are these present in the dialog.tlk of BG1, but not in the one from BG2, because the slots have been repurposed for BG2?
• (Not a question, just trying to be thorough:)
  • Starting at 30958, there are references to "Keelor the Dwarf"; these are from NTotSC in my installation

The default tlk has a ton of redundancies, yes--easiest example is that pretty much every creature has the same tooltip and name, but these are different strings. The Minsc creature files use "Minsc" (9501) as his name and "Minsc" (9482) as his tooltip. 

IIRC WeiDU uses the last match of a string when determining which one to use when a new string is requested, but that's from memory. If it's not the last, it's first. (Helpful, I know.)

The broader questions bout EET string handling I'm not to sure since I don't know the EET internals. There are a myriad of ways you could force WeiDU to add redundant strings, it's just that you'd have to go out of your way to do it. EET intentionally makes the oBG strings be EET+200000 for mod interoperability, regardless of redundancy.

55 minutes ago, Lurker said:

Which is another reason why I don't rely on uninstalling - only in special cases and when close to the "onion's skin". Scripted reinstalls ftw.

To be fair, strings are really the only place that WeiDU behaves a little strange, everything else is rolled back as you expect.

That doesn't mean there can't be wonk depending on how a mod is made. WeiDU allows modders to directly access the shell, which means you could make a bunch of changes without WeiDU 'knowing', meaning WeiDU won't track and roll back such changes. (This was very commonly done when installing audio so that we could run sox/oggdec. This is till done via shell, but handled through a ubiquitous function that lets WeiDU track changes.). There are also options to make some changes that intentionally won't be uninstalled--e.g. I use the copy-no-track to keep some reference material for Tweaks to in a weidu_external folder so that items end up in the same place during a reinstall.

1 hour ago, Lurker said:

Err... Does that mean I could have just copied over my dialog.tlk from my previous installation (v1) to my template for v2, and then just applied the slightly altered payload file?    (Need to check this out.)

Perhaps. However, it's entirely possible that mod A adds strings and then mod B alters them. If you start over with this tlk, in such a case mod A would add the (unaltered) strings again (since they no longer match existing strings) and mod B would alter them, again. An easy example of this is various Tweaks components alter item descriptions--Rebalanced Proficiencies alters descriptions (including those from mods) to match the changes it makes to items.

 

[Lurker]

51 minutes ago, CamDawg said:

1 hour ago, Lurker said:

Err... Does that mean I could have just copied over my dialog.tlk from my previous installation (v1) to my template for v2, and then just applied the slightly altered payload file?    (Need to check this out.)

Perhaps. However, it's entirely possible that mod A adds strings and then mod B alters them. If you start over with this tlk, in such a case mod A would add the (unaltered) strings again (since they no longer match existing strings) and mod B would alter them, again. An easy example of this is various Tweaks components alter item descriptions--Rebalanced Proficiencies alters descriptions (including those from mods) to match the changes it makes to items.

So, I did another reinstall,  with the dialog.tlk from v1 copied over to v2 before starting the installation. Looks like a 100% success so far, no need to update anything in my saves.

dialog.tlk grew ~ 150 KB, but since I also added a larger quest mod and some smaller stuff compared to v1, that's expected.

Thanks again @CamDawg, without your input, I wouldn't have tried this.

[Lurker]

I've updated my initial comment and the attached files.

Also, here is a patch available to make the component to restore the reputation increase sound effect from "The Artisan's House Rule Tweaks" again installable on EET. Since I edited this an previously, it might have gone mostly unnoticed, especially regarding this:

On 10/28/2023 at 2:36 PM, Lurker said:

if providing this here causes complaints either by the original author or the G3 staff, I'll remove the public patch, naturally.

Pinging @The Artisan.