[Feedback needed] merging TLK files or not

[K4thos]

Out of curiosity, why reserve up to 103,499? Why not start the BGEE strings at a nice round number like 200,001? Then any changes modders have to make to BG1-specific strings can simply be changed to add a 2 and some zeros before the old string.

 

I know that's kind of trivial, but that kind of trivial standardization can actually be helpful to modders. (Also it would leave tons of room for the small chance that Beamdog does add some kind of content expansion to BG2EE.)

 

Is there a performance or other reason to try to keep the strref range low?

dunno. Seems odd to add so much junk just for aesthetics. It would indeed ensure that SoD like expansion for BG2:EE wouldn;t force us to change starting point. It's rather unliekly situation to happen though. Opinions?

[argent77]

I'm also in favor of adding a generous amount of filler entries to the talk table. Dummy entries don't take up too much space (26 bytes per entry) and the first reserved strref for user-defined strings is 3,000,000. And, as unlikely as it sounds, Beamdog might still solve Skie's fate in BG2EE with a mini expansion.

[GrimLefourbe]

I'm not sure I understand the point of the dummy entries, is the size of bg2 tlk hard-coded into EET rather than read on the fly ?

 

On the other hand I don't see how it could hurt if the tlk lookup time isn't hugely impacted by the number of entries, which seems to be the case.

[Jarno Mikkola]

I'm not sure I understand the point of the dummy entries, is the size of bg2 tlk hard-coded into EET rather than read on the fly ?

Read the first post. K4thos said the reasons there.

[K4thos]

I'm not sure I understand the point of the dummy entries, is the size of bg2 tlk hard-coded into EET rather than read on the fly ?

 

On the other hand I don't see how it could hurt if the tlk lookup time isn't hugely impacted by the number of entries, which seems to be the case.

Current BG2:EE with latest patch has 103213 TLK strings. Let's say we merge whole BG:EE TLK on top of it starting from 103214. Now the EET saves for vanilla content works for everybody (if you send me it I will have correct journal entries, dialogues, names etc even though I have entirely different mods and use different language - the game works exactly the same as official games in this aspect). Now let's say patch 3.0 adds 100 new strings to BG2:EE TLK. In such case we would need to move the starting point of BG:EE TLK file in EET further if those 100 strings haven't been reserved, which would make all old saves no longer compatible with new installation.

[smeagolheart]

There's no way to have a bgdialog.tlk and a dialogue.tlk and just reference one or the other is there?

[Jarno Mikkola]

There's no way to have a bgdialog.tlk and a dialogue.tlk and just reference one or the other is there?

Yeah, with an .exe, it's easy... but if you wish to avoid that, you might want to combine the two and have a one solid game... like for reals.

[subtledoctor]

There's no way to have a bgdialog.tlk and a dialogue.tlk and just reference one or the other is there?

Interesting. Isn't that what argent77's DLC-Builder tool is for? Why not keep the BGEE .tlk file inside a .zip pseudo-DLC archive.

 

(Answer, probably: that would be a totally legit way to do things, if EET hadn't already been in development for a year when that function became available. But to do that now would more or less mean starting over from scratch.)

[Jarno Mikkola]

(Answer, probably: that would be a totally legit way to do things, if EET hadn't already been in development for a year when that function became available. But to do that now would more or less mean starting over from scratch.)

More probable is that you need to undo the whole thing if you wish to further modify the EET as a game, so just lob the 2.5GB's in addition to the mod install size, which would put everyone except the lone wanderer that has never heard of tweaks packs to that boat.

Ever heard of how the SoD made the whole game unmod-able, without the modmerger. See now ?

 

Yes, if the weidu can be made to use the archived data as the game data then perhaps we'll be talking about this ... but until that point, you'll be dreaming man.

Ouh, and one should never pack the dialog.tlk into that file... but then leaving it out will make the archived data only able to use the existing data already in the games dialog.tlk, which is really hard to do too.

[K4thos]

 

There's no way to have a bgdialog.tlk and a dialogue.tlk and just reference one or the other is there?

Interesting. Isn't that what argent77's DLC-Builder tool is for? Why not keep the BGEE .tlk file inside a .zip pseudo-DLC archive.

 

(Answer, probably: that would be a totally legit way to do things, if EET hadn't already been in development for a year when that function became available. But to do that now would more or less mean starting over from scratch.)

 

not sure what you mean. DLC-Builder builds SoD like DLC files which means the same thing you have in your game folder but zipped (preferably with only things related to expansion in it). As far as I know TLK inside DLC file completely replaces the one in main lang directory. Same with 2DA and all other files. That was the reason why SOD dlc had to be unpacked before you could add anything to the TLK files even if the mod was designed only for vanilla BG:EE. Either way in EET you want for the dialogue stuff to be always available, even when you go back to Naskeel from the end of ToB.

[Jarno Mikkola]

I do have a legit concern about combining the .tlk's, and that's modified BG1 game, as you know, you have made the BG1EE mods compatible with EET by making them install-able before the conversion. Now if one has a vanilla BG1EE, and the other has hundreds of small but still legit mods installed on their BG1EE game, it's going to make a severe differential to the .tlk conversion process. Let alone the cross game resources etc, aka a mod brings a BG2EE item to BG1EE and so edits it to fit, which one will be the victor in override, or do you again go and make a feature. Can you even automate the whole process ? And that's the concern.

Yes, one can convert the .tlk, but you need to also automate the file string reference updates, are you going to make a whole TLK-to-TLK cross string comparison feature ? I just wouldn't want to be you.

[K4thos]

Now if one has a vanilla BG1EE, and the other has hundreds of small but still legit mods installed on their BG1EE game, it's going to make a severe differential to the .tlk conversion process

 

what conversion process? You mean this?

1. Strings added by mods installed on BG:EE are at the end of BG:EE TLK file (nothing has to be changed in those mods, that's how weidu works)

2. BG:EE TLK file is appended at the end of BG2:EE TLK file

3. During installation some BG:EE string references in imported resources are automatically changed to BG2:EE ones just to make the descriptions more consistent (general item descriptions etc.). Also opcodes 139 and 267 will use values from BG2:EE if present based on pre-made array (automatically detected by comparing string numbers). On top of that additional macro adds more opcode 267 effects if BG2 expects more strings to be immune to when particular opcode is used compared to BG:EE.

4. During installation if the string was not swapped in previous step than it's get updated with appropriate value added to original value (so they end up being always the same for vanilla resources between different installations).

5. Mods installed after EET appends at the end of output TLK file (nothing has to be changed in them, that's how weidu works)

 

Yes, one can convert the .tlk, but you need to also automate the file string reference updates, are you going to make a whole TLK-to-TLK cross string comparison feature ? I just wouldn't want to be you.

 

no idea what you mean, sorry. Exactly which reference updates you refer to? String references present in BG:EE resources? If yes than of course they would be updated during installation. Currently EET adds @ or uses SAY / RESOLVE_STR_REF. In next RC it will simply add particular number (total number of strings in BG2:EE + some dummy junk to reserve place for future official BG2:EE TLK changes) to existing string value. But installed mods doesn't affect it at all, so what's the problem?

[AL|EN]

 

One thing that springs to mind if you do merge the TLK files is that the string references may not be completely static if you append the BGEE ones after BG2EE - there's the potential that a new patch for BG2EE could add additional strings. Still, I like the idea.

That's indeed a valid concern considering patch 2.x added 167 new strings to BG2 (related to new kit, spells, story / legacy of bhaal modes and item importation from SoD). Currently BG2:EE with latest patch has 103213 TLK strings. In order to ensure future compatibility with saves we would need to assign some dummy entries at the end of file to reserve strings for future official changes in BG2:EE TLK. Not sure how many though. If there won't be SoD like expansion released for BG2:EE than I think we could automatically reserve strings for let's say up to 103499. So BG:EE TLK file would always start from string 103500. Or we could reserve even more if 286 is not safe enough prediction of what may be needed in future by BG2:EE.

 

Make BG:EE starting string as 110000, much safer than ~300.

[K4thos]

BG:EE TLK actually starts from 200 000 as suggested by subtledoctor. And those dummy entries weight next to nothing as mentioned by argent77. In fact the output file is 2MB smaller than I predicted in the first post. And it can become even more comparable to TLK size from old implementation if we replace unused BG:EE strings with dummy entries (I can write a code that checks if particular string is used in game and generate an array that will do the job during installation). Although this has low priority considering it doesn't really change anything other than size of TLK.

 

Mini f.a.q:

1. So wait, you're telling me there is like 96 000+ reserved entries before BG:EE TLK and new entries from mods start?

Yep, that's the case

2. How much time it takes to generate such file?

Just a few seconds. There are some weidu quirks reported here but I've manged to implement workarounds.

3. What will you do with so much junk entries?

Well, why not extend the persistent save feature also for IWD-in-EET content? Here are TLK ranges (if you install IWD-in-EET, otherwise strings between BG2:EE and BG:EE TLK will be empty):

- BG2:EE - 0 (currently in TLK: 103 213, reservation for up to 111 000 strings)

- IWD:EE - 111 000 (currently in TLK: 40 851, reservation for up to 44 000 strings)
- IWD2 - 155 000 (currently in TLK: 41 421, reservation for up to 44 000 strings)

- EET Worldmap - 199 000 (reservation for up to 1 000 strings)
- BG:EE - 200 000 (not restricted, so also supports mods installed on BG:EE before appending, which makes mod stings attached at the end of TLK)
- all mods not restricted, starting at the end of appended BG:EE TLK file

Of course if Beamdog will release some BG2:EE expansion and those 7787 reserved strings won't be enough for it than we will need to drop save persistence for IWD-in-EET content in order to free more strings for BG2:EE - not really a problem.

 

I'm testing it right now and don't see any problems with this implementation. If someone has different opinion regarding it - please share your concerns.