Foreign lang. mod displays nonsense - any way to fix?

[Guest RedRaven]

Hello-

I have an item mod that was written in  Russian, but doesn't include translation. So my item descriptions look like this:

Âîçìîæíîñòè ýêèïèðîâêè:
- +50% Ñîïðîòèâëåíèÿ Ìàãèè
- Èììóíèòåò ê Îòêà÷èâàíèþ Óðîâíåé

The mod does come with an English spoiler file with item descriptions, but it's a pain to be constantly having to look them up on my printed list instead of having them just display correctly in the game. I was considering trying to edit the descriptions (just for my own convenience). I could type them in from the descriptions in the spoiler file, which would be adequate, but not ideal. I can tell from looking at the gibberish descriptions that they include more text than is in the file. The file is more of a summary than in the format of a proper description. So, I would prefer to have the descriptions the mod author intended.

Is there any way to do that? I assume that the issue is that it was written with a Russian keyboard and I have an English system. It's like the Russian got translated into some sort of placeholder characters since my system doesn't understand Russian. And I need to translate those characters into English or normal Russian that I can then translate.

Any thoughts?

Thanks! - RR

 

[jastey]

This is because the editor you are using can't translate the cyrillic script. I use Notepad++ with installed Russian localization, then the texts are shown correctly.

EDIT: Then I'd run it through DeepL online translator.

[ORedRaven04]

Wonderful! Thank you so much, jastey, and for such a quick reply. I figured I was just going to be told I was out of  luck. I will give that a try.

Much appreciated!

Edit: Do I have to install Notepad++ as if I am Russian, or is there a way to install in English and add Russian as an option? I've scanned the Notepad++ help info, but I'm not seeing an answer.

Edited June 16, 2023 by ORedRaven04
Follow-up question

[Graion Dilach]

One of the settiings within Encoding -> Character sets -> Cyrillic should be the solution.

[jmerry]

Ah, text encodings.

The original ASCII was designed by English speakers, and encoded that particular version of the Latin alphabet. Then things went international, and that wasn't enough. Other Latin-alphabet languages needed letters with various marks. Russian needed a whole new alphabet. Chinese and Japanese needed their own character sets, and so on. ASCII has some unused slots, and you can extend the system - but space is finite. So standards proliferate. One encoding uses the empty spaces for all those letters with accents and umlauts and so on that various Latin-alphabet languages need. One uses them for the Cyrillic alphabet, so you can write Russian with it. And when you try to read text with the wrong encoding, you get something like the gibberish posted above - that's clearly a case of text using a completely different character set being read in an encoding meant for variations on the Latin alphabet.

The modern standard is Unicode - one encoding for the entire world, at least in theory. Achieved by allowing characters to use multiple bytes. And the lifespan of these games spans the transition from various extended-ASCII encodings to Unicode; the EEs use UTF-8 to encode all their text, while the original games used different encodings based on localization. Mods which cover both original and EE games have to covert text between different encodings to handle that.

As for reading stuff in a text editor... there should be some sort of preferences for what text encoding to use. At least when opening or saving files. Details vary based on which program you're using. I rooted around in my own (TextEdit, the built-in Mac OS editor) and found a drop-down menu of ten possible encodings. None of which were "Cyrillic" or "Russian". Hit the "Customize Encodings List" option at the bottom of that, get a menu of over a hundred encodings to either include or not include on that short list. Oof.

[jastey]

48 minutes ago, ORedRaven04 said:

Do I have to install Notepad++ as if I am Russian, or is there a way to install in English and add Russian as an option? I've scanned the Notepad++ help info, but I'm not seeing an answer.

You install it in English, and one install step is the choice of which localizations / letter formats you want to add. If you tick the box for Russian there, Notepad++ will be able to display it correctly.

[ORedRaven04]

Thank you, Graion. I do see that. I can understand how that would work if I was wanting to encode something, but not how to de-code. (I'm not really a code person, I just know how to poke around in Near Infinity a bit when a mod is misbehaving.) If I open the item file that way, it looks like I'm getting more definition stuff, like i can see bits and pieces of English words like "Belt01" and things... but I'm not sure how to get to the actual description text. I can copy it from Near Infinity, but then I don't know what to do with it from there.

Thank you jmerry. I did not plan to learn all about language encoding today, but now I'm down the rabbithole. I have been poking around and trying various programs and various types of encoding, but the best I can get is those bits and pieces of a few English words. It doesn't seem to be using any of the standard encoding types.

[jastey]

I was assuming you have the mod files that can be opened by an text editor, i.e. the .d-files, and .baf, .tra, .tp2 etc.

If you have the scrumbled symbols in NI then something went wrong in the installation? NI should be able to display Russian texts.

[ORedRaven04]

Hmmm. Well, I tried just opening the files themselves (from the mod install .zip that I unzipped, rather than my game directory)... and those do give me English, but the descriptions are all goofy.  For example "Identified Description: Amber, look. It was fun while it lasted, but don't you think that we should go on our separate paths?"

The ones in my game directory appear to have actual item descriptions, though they aren't readable. I can see the structure though, and things like 50% and THACO: +5, etc. And the items do seem to do what they are supposed to do. So I just need to fix whatever is going on with the encoding.

It's possible my install is bad. I did have to uninstall and reinstall something part way through and that never seems to go well. I'm also missing a couple of mods completely, even though I theoretically installed them. (Which is weird because I've played with these same mods before and the same install order. So, it's probably due to that mistake in the process). I'll normally start over if I have any problems rather than trying to uninstall and reinstall, but my external drive with my game install files decided to try to die and had to go to the shop and I couldn't get into my GOG account because it refused to send me an authentication mail.... *tears out hair*  Both of those things should be fixed now, but that doesn't help me currently.

I guess I will just finish my current play-through and put up with reading descriptions off my printout and try a clean install afterward. Something definitely seems messed up.

Thank you all for the help. I have learned something new today about language encoding at least.

Edited June 16, 2023 by ORedRaven04
Additional info

[jmerry]

14 minutes ago, ORedRaven04 said:

If I open the item file that way, it looks like I'm getting more definition stuff, like i can see bits and pieces of English words like "Belt01" and things... but I'm not sure how to get to the actual description text. I can copy it from Near Infinity, but then I don't know what to do with it from there.

The game text is all (well, nearly all) stored in dialog.tlk and referenced with string reference numbers. For example, the "General description" field for BELT01.ITM in BGEE is simply the number 6466. The description text for that item is then string #6466.

Mods generally add new text to the game through WeiDU operations, extending dialog.tlk with new entries that have larger numbers. Then those numbers get referenced in the various in-game places the text is needed.

Well-behaved mods separate out the text they're adding into .tra files - the same set of files for each supported language. Then the mod code uses numbered references to those tra files when any text is being added. Those tra files should be encoded in something appropriate to the language they're in, and any character set conversions needed are done at installation time.

...

4 minutes ago, ORedRaven04 said:

Hmmm. Well, I tried just opening the files themselves (from the mod install .zip that I unzipped, rather than my game directory)... and those do give me English, but the descriptions are all goofy.

This is a mismatch error; text added by one mod (part of that mod install zip) gets a number, and a different mod install (what you did with that Russian-language item mod) uses that same number for something else. When you open the file alone in NI, any string references get interpreted based on what the current install's dialog.tlk says, not whatever the lone file was built with.

[ORedRaven04]

9 hours ago, jmerry said:

The game text is all (well, nearly all) stored in dialog.tlk and referenced with string reference numbers. For example, the "General description" field for BELT01.ITM in BGEE is simply the number 6466. The description text for that item is then string #6466.

Mods generally add new text to the game through WeiDU operations, extending dialog.tlk with new entries that have larger numbers. Then those numbers get referenced in the various in-game places the text is needed.

Well-behaved mods separate out the text they're adding into .tra files - the same set of files for each supported language. Then the mod code uses numbered references to those tra files when any text is being added. Those tra files should be encoded in something appropriate to the language they're in, and any character set conversions needed are done at installation time

Aha, thank you, jmerry. That makes sense. I could tell that all the information wasn't always contained with the item file itself, but didn't understand how it was structured and where the info might be stored. Interesting.

9 hours ago, jmerry said:

This is a mismatch error; text added by one mod (part of that mod install zip) gets a number, and a different mod install (what you did with that Russian-language item mod) uses that same number for something else. When you open the file alone in NI, any string references get interpreted based on what the current install's dialog.tlk says, not whatever the lone file was built with.

I'm very confused by that because I'm 99% positive I've played with these same mods in the past without any big issues / conflicts. I will have to double check for those string references and see if they are being used by multiple mods, but I think something went really wrong with my install. I know theoretically you can uninstall and reinstall stuff on the fly, but I've had issues every time I've done that.

The place where I'm seeing that mixed up text isn't even in my install, though. It's in the straight mod file of the newer version. I downloaded a newer version (5.0.5 as opposed to the 5.0.4 I'm currently playing with) just to take a look at. I unzipped it in a location having nothing to do with my BG2 directory and opened the item file with NI using "open external file". So, it shouldn't be having any involvement from my current install. Very strange.

[jastey]

1 hour ago, ORedRaven04 said:

So, it shouldn't be having any involvement from my current install. Very strange.

It does, because you specified a game directory when opening NI. Otherwise, you wouldn't be able to browse files with it. So, NI takes the string references in the dialog.tlk of the game you opened NI with, even though the file you opened is from elsewhere.

Try to read the item description in the other mod files, where the texts are in that get assigned to the items upon install. They should be in the modname.tp2, or in a "translation" or "tra" subfolder in .tra files. These you can open with a text editor.

[ORedRaven04]

6 hours ago, jastey said:

It does, because you specified a game directory when opening NI. Otherwise, you wouldn't be able to browse files with it. So, NI takes the string references in the dialog.tlk of the game you opened NI with, even though the file you opened is from elsewhere.

Try to read the item description in the other mod files, where the texts are in that get assigned to the items upon install. They should be in the modname.tp2, or in a "translation" or "tra" subfolder in .tra files. These you can open with a text editor.

Ahhh, okay that makes sense. I guess I'm stuck until I finish this game and reinstall then.

Yes, I found the English .tra and was able to open it with Notepad++. So the correct stuff is there; it just seems like something went screwy with my install or I have a conflict. I'll toy with it in a few days when I've finished my current game.

[jastey]

If you have the scrumbled symbols in your game then the text format could be wrong. The EE games need texts in utf8 (without BOM) but the classic engine in ANSI (or whatever format this is for cyrillic scrpt). The format could be wrong if e.g. a mod that is not updated for the EEs is installed to BG(II):EE with the wrong text format.

[ORedRaven04]

2 hours ago, jastey said:

If you have the scrumbled symbols in your game then the text format could be wrong. The EE games need texts in utf8 (without BOM) but the classic engine in ANSI (or whatever format this is for cyrillic scrpt). The format could be wrong if e.g. a mod that is not updated for the EEs is installed to BG(II):EE with the wrong text format.

Hm. I assume the opposite would be true as well, then? I'm actually playing vanilla BG2. (I haven't quite decided if EE appeals to me or not.) I can always try one of the older version, before it was updated for EE compatibility. It doesn't look like the most recent changes are big content changes, just fixing spelling mistakes and adding another language, etc,

I will have to look through all of my mods and see if they are all updated for EE. I am kind of curious to try the new classes and all, and it seems like everyone has mostly migrated to EE now, but I have heard mixed things about it and I'm so used to the old version.