Possible Encoding BUG in 5.1 + Possible Improvement

[Frenzgyn]

@CamDawg@jastey

Since Amber is not on G3's github, it's a bit clumsy to do bug reports or propose pulls, I usually don't like to ping people, but I assume you two are the mantainers of the mod, so sorry for this exception.

1) BUG (possible leftover of 5.0 encoding-odditiy fixed in 5.1?):

Many (if not all) item descriptions have the " – " added. It seems and encoding error. The culprits seem to be english tra.

That's what happens in game:

 

2) In EEs the artworks of quest NPCs (and Vampirized Amber) look *REALLY* awful, since the engine upscale the tiiiny res images created with oBG2 in mind.
I've took the time to do a bit of resampling/upscaling.
Far - ETREMELY FAR - from perfect, but at least they don't make my eyes bleed anymore, so yeah, I suppose it could be considered an improvement.
I've done the same to Amber's portraits, they always looked a bit too blurry to me, but it's barely noticeable.
Here is a comparison of the various resolution. In game the smaller pictures looks just the same, the bigger ones are just a tiny bit sharper.

And here are the "NPCs", the reason I did all the stuff (sorry Michelle Yeoh, don't have the guts to rework it from the source image ).

 

I've them properly renamed and packed HERE,

dropping the three npc files in "ee" folder will just overwrite the low res version during installation on EEs together wit item images, so no change to setup or folder structure is needed. Amber's portait could just be overwritten (or not) by the slightly reworked version.

I don't know if this could be integrated in the mod when/if a new version will be released, it's not strictly a "technical update", but it's not *really* a content change either, I tried to preserve the original images (sorry again Michelle), so no - at least - no harm done in sharing, I think. If there are any problems or issues about it, I'll take down the stuff, just let me know.

Edited November 15, 2023 by Frenzgyn

[jmerry]

So that encoding error – it's clearly meant to be the long dash character. Which is hard to visually distinguish from the short dash character - but the latter is in the ASCII set that's the same in UTF-8 and the pre-EE encoding, and the former isn't so it can get scrambled.

Edited November 14, 2023 by jmerry

[Frenzgyn]

Sorry if it is a stupid comment, but the english traified files (ee, dialogs) are already in UTF-8, with the en dash, after the installation it got the odd ANSI equivalent –. Spanish and German ee and dialogs are in ANSI.
This happens because the two file are read assuming to be ANSI and then converted back to UTF-8?
That's the codeblock:

    ACTION_IF GAME_IS ~bg2ee eet~ BEGIN
      OUTER_SET ee_game = 1
      // convert strings to UTF-8 for BGEE/BG2EE
      ACTION_DEFINE_ARRAY cdreload BEGIN dialogs ee END
      ACTION_DEFINE_ARRAY cdnoconvert BEGIN setup END // List of tra files that contain ONLY strings for the WeiDU installer and NOT game content
      LAF HANDLE_CHARSETS
    INT_VAR
      infer_charsets = 1
    STR_VAR
      tra_path = ~amber/tra~
      noconvert_array = cdnoconvert
      reload_array = cdreload
       END
    END ELSE BEGIN

 

I suppose it is related to this

Now, if it is the case how should it be fixed in a solid way?

a) Converting english ee.tra into ANSI and let weidu convert it back to UTF-8 when installing on EE/EET?

Or

b) Purging ee.tra, replacing the various "–", "é" (in "protégés") and "ō" (in "Ninjatō") with something else, for example keeping the oBG2 item description "Nija-to"? 

"dialogues.tra" is in UTF-8, but have no symbol that create issues.

Sheesh. I HATE this stuff.

Edited November 15, 2023 by Frenzgyn

[jmerry]

Or to convince the system to not do any conversion on the file that's already in UTF-8? Conceptually, that shouldn't be hard ...

But then, I mod for EE only. All I have to do for my own stuff is ensure that it's all in UTF-8.

[Frenzgyn]

2 hours ago, jmerry said:

Or to convince the system to not do any conversion on the file that's already in UTF-8? Conceptually, that shouldn't be hard ...

But then, I mod for EE only. All I have to do for my own stuff is ensure that it's all in UTF-8.

It seems the issue was caused by doing BOTH things jastey suggested in the thread I linked, not one OR the other.
All "ee.tra"s already in UTF-8 --> exclude ee.tra fron conversion. (being text used only in EEs, no backporting issues).
All "ee.tra"s in ANSI --> let weidu convert them all.

Weidu is not a place for compromise .

By the way STILL HATE all the encoding/keyboard layouts and what the like stuff.
Sorry for al the fuss.

[CamDawg]

And the text encoding is fixed in v6.

I had this thread bookmarked as an issue for the next version, and I realize that while I was focusing on the text encoding issue, I completely missed the upscaled portraits. I'll make a note for v7, sorry @Frenzgyn.