translation [Tutorial] How to properly handle mod translation by using HANDLE_CHARSETS function, with AUTO_TRA

[AL|EN]

EDIT: Turns out the code is incompatible with AUTO_TRA. Until future investigation, you can't use this if you are not wiling to resign from using AUTO_TRA function.

EDIT: Guide was updated to reflect recent WeiDU 249 changes. AUTO_TRA is now fully supported.

Introduction:

Since introduction of HANDLE_CHARSETS, modders have given the ability to support translation for both classic and enhanced editions of the Infinity Engine games.
The function itself was doing it's job: convert file encoding from one to another. But it also had annoyances which various modders was able to overcome via custom code.
Now, with the introduction of `out_path` parameter, all of the annoyances are gone. This is an attempt to show how to get the best out of HANDLE_CHARSETS.

How it was handled before:

1. translators provide .tra files using their own preferred local encoding
2. HANDLE_CHARSETS function convert those files into UTF8 encoding when mod was installed on enhanced game editions

How it is handled now:

1. translators provide .tra files using UTF8 encoding
2. when the mod is installed on classic game editions, the temporary copy of translation files are converted into ASCII encoding so the source files are not touched
3. no modification of original UTF8-files when mod is installed on enhanced game editions

Using new way will save you from annoyances and allow for new features:

• modders no longer need to watch-out to not break custom encoding when editing foreign tra files
• modders can receive translations via forum posts and it can be used instantly, without hustle 
• translators are no longer required to send files just to update one single line
• translation can be provided by editing files directly via GitHub online interface
• DOS/Shell console messages can finally have localized characters displayed properly
  (required Windows 10 1903 and above, for Windows 7/Windows 8.1 you have to use Cmder/ConEmu/other modern terminal with "chcp 65001" command)

Requirements:

• required minimum version of WeiDU is 249
• if you are updating old mod, you have to properly convert .tra files to UTF8 using correct codepage for specific language

Full code example: https://github.com/InfinityMods/WeiDU-UTF8

Spoiler

 BACKUP "weidu_external\backup\WeiDU-UTF8-Example" // move backup folder outside of the mod data folder, %MOD_FOLDER% variable will be fine
SUPPORT "https://www.gibberlings3.net/forums/topic/32369-tutorial-how-to-properly-handle-mod-translation-by-using-handle_charsets-function-with-auto_tra"
VERSION "3.0.0"

ALWAYS

// do not convert Setup.tra files, those should be UTF8 already
    ACTION_DEFINE_ARRAY arrayNoConvert BEGIN Setup END 
    ACTION_DEFINE_ARRAY arrayReload BEGIN Mod NPC NPCP NPC25P END
    
    // define variable for HANDLE_CHARSETS out_path so it can be reused later
    OUTER_TEXT_SPRINT "HANDLE_CHARSETS_OUT_PATH" EVAL "weidu_external\Lang\%MOD_FOLDER%"
    
    LAF HANDLE_CHARSETS
        INT_VAR
            from_utf8 = 1
            infer_charset = 1
            verbose = 1 // you can disable this if you want to skip verbose output
        STR_VAR
            default_language = english
            tra_path = EVAL "%MOD_FOLDER%\Lang"
            out_path = EVAL "%HANDLE_CHARSETS_OUT_PATH%" // location of converted .tra files
            noconvert_array = arrayNoConvert
            reload_array = arrayReload
    END
    
    // for AUTO_TRA: if the game is the enhanced edition, the conversion will not happen so all .tra files should be loaded from "%MOD_FOLDER%\Lang"
    ACTION_IF (GAME_IS "bgee bg2ee eet iwdee pstee") THEN BEGIN
        OUTER_TEXT_SPRINT "TRA_LOCATION" EVAL "%MOD_FOLDER%\Lang"
    END

    // for AUTO_TRA: if the game is the classic edition, all converted .tra files will be loaded from HANDLE_CHARSETS out_path
    ACTION_IF NOT (GAME_IS "bgee bg2ee eet iwdee pstee") THEN BEGIN
        OUTER_TEXT_SPRINT "TRA_LOCATION" EVAL "%HANDLE_CHARSETS_OUT_PATH%"
    END

END

// for the EE games, the variable will point to the untouched .tra files
// for classic games will point to the HANDLE_CHARSETS out_path value where converted files are stored
AUTO_TRA "%TRA_LOCATION%\%s"

LANGUAGE "English" "English" 
    "%MOD_FOLDER%\Lang\English\Setup.tra"
    "%MOD_FOLDER%\Lang\English\Mod.tra"
    "%MOD_FOLDER%\Lang\English\NPC.tra"
    "%MOD_FOLDER%\Lang\English\NPCP.tra"
    "%MOD_FOLDER%\Lang\English\NPC25P.tra"

LANGUAGE "Czech" "Czech"
    "%MOD_FOLDER%\Lang\Czech\Setup.tra"
    "%MOD_FOLDER%\Lang\Czech\Mod.tra"
    "%MOD_FOLDER%\Lang\Czech\NPC.tra"
    "%MOD_FOLDER%\Lang\Czech\NPCP.tra"
    "%MOD_FOLDER%\Lang\Czech\NPC25P.tra"

LANGUAGE "French" "French"
    "%MOD_FOLDER%\Lang\French\Setup.tra"
    "%MOD_FOLDER%\Lang\French\Mod.tra"
    "%MOD_FOLDER%\Lang\French\NPC.tra"
    "%MOD_FOLDER%\Lang\French\NPCP.tra"
    "%MOD_FOLDER%\Lang\French\NPC25P.tra"

LANGUAGE "German" "German"  
    "%MOD_FOLDER%\Lang\German\Setup.tra"
    "%MOD_FOLDER%\Lang\German\Mod.tra"
    "%MOD_FOLDER%\Lang\German\NPC.tra"
    "%MOD_FOLDER%\Lang\German\NPCP.tra"
    "%MOD_FOLDER%\Lang\German\NPC25P.tra"

LANGUAGE "Italian" "Italian" 
    "%MOD_FOLDER%\Lang\Italian\Setup.tra"
    "%MOD_FOLDER%\Lang\Italian\Mod.tra"
    "%MOD_FOLDER%\Lang\Italian\NPC.tra"
    "%MOD_FOLDER%\Lang\Italian\NPCP.tra"
    "%MOD_FOLDER%\Lang\Italian\NPC25P.tra"

LANGUAGE "Polish" "Polish"
    "%MOD_FOLDER%\Lang\Polish\Setup.tra"
    "%MOD_FOLDER%\Lang\Polish\Mod.tra"
    "%MOD_FOLDER%\Lang\Polish\NPC.tra"
    "%MOD_FOLDER%\Lang\Polish\NPCP.tra"
    "%MOD_FOLDER%\Lang\Polish\NPC25P.tra"

LANGUAGE "Russian" "Russian" 
    "%MOD_FOLDER%\Lang\Russian\Setup.tra" 
    "%MOD_FOLDER%\Lang\Russian\Mod.tra"
    "%MOD_FOLDER%\Lang\Russian\NPC.tra"
    "%MOD_FOLDER%\Lang\Russian\NPCP.tra"
    "%MOD_FOLDER%\Lang\Russian\NPC25P.tra"

LANGUAGE "Spanish" "Spanish"
    "%MOD_FOLDER%\Lang\Spanish\Setup.tra"
    "%MOD_FOLDER%\Lang\Spanish\Mod.tra"
    "%MOD_FOLDER%\Lang\Spanish\NPC.tra"
    "%MOD_FOLDER%\Lang\Spanish\NPCP.tra"
    "%MOD_FOLDER%\Lang\Spanish\NPC25P.tra"

BEGIN @1001 DESIGNATED 1001 LABEL "WeiDU-UTF8-Example-Main"
    REQUIRE_PREDICATE GAME_IS ~bgee tob bg2ee bgt eet~ "Supported games: bgee tob bg2ee bgt eet"

    COPY "%MOD_FOLDER%\Characters\NPC.cre" "Override"

    SAY NAME1 @11001
    SAY NAME2 @11001
    SAY BIO @11002
    SAY INITIAL_MEETING @11002
    WRITE_ASCII 0x2cc "NPC" #8
    WRITE_ASCII 0x280 "NPC" #32

    COMPILE
	"%MOD_FOLDER%\Dialogs\NPC.d"
    "%MOD_FOLDER%\Dialogs\NPC25P.d"
    "%MOD_FOLDER%\Dialogs\NPCP.d"

    APPEND "pdialog.2da" "NPC NPCP NPCJ NPCP NPC25P NPC25J NPC25D NPC25S" UNLESS "NPC"
    APPEND "interdia.2da" "NPC BNPC BNPC25" UNLESS "NPC"
    
    ACTION_IF GAME_IS "bgee" BEGIN
        STRING_SET 15594 @10001
        EXTEND_TOP "AR2600.bcs" "%MOD_FOLDER%\Scripts\AR2600.baf"
    END

    ACTION_IF GAME_IS "bgt" BEGIN
        STRING_SET 11676 @10001
        EXTEND_TOP "AR0015.bcs" "%MOD_FOLDER%\Scripts\AR0015.baf"
    END

    ACTION_IF GAME_IS "tob bg2ee" BEGIN
        STRING_SET 70742 @10001
        EXTEND_TOP "AR4000.bcs" "%MOD_FOLDER%\Scripts\AR4000.baf"
    END

    ACTION_IF GAME_IS "eet" BEGIN
        STRING_SET 215594 @10001 // +200000
        EXTEND_TOP "BG2600.bcs" "%MOD_FOLDER%\Scripts\BG2600.baf"
    END

 

Code explanation:

 BACKUP "weidu_external\backup\WeiDU-UTF8-Example" // move backup folder outside of the main mod data folder

Move backup folder outside of the main mod data folder. The %MOD_FOLDER% is read from this value but WeiDU 249 should correct it to the main mod folder name.

// do not convert Setup.tra files, those should be UTF8 already
ACTION_DEFINE_ARRAY arrayNoConvert BEGIN Setup END 
ACTION_DEFINE_ARRAY arrayReload BEGIN Mod NPC NPCP NPC25P END

Define two arrays for `noconvert_array` and `reload_array` parameters of `HANDLE_CHARSETS` function because those parameters require array. You cannot provide the filenames directly. The .tra file extension is optional. Do not convert Setup.tra files, those should be UTF8 already.

LAF HANDLE_CHARSETS
    INT_VAR
        from_utf8 = 1 //the source tra files of all languages are UTF8
        infer_charset = 1 //set the correct encoding for classic games based on the detected game and associated encoding
        verbose = 1 //display more debugging  info

Parameter from_utf8 = 1will tell WeiDU that the source tra files of all languages are UTF8. The infer_charset = 1 will set the correct encoding for classic games based on the detected game and associated encoding. The verbose = 1 will display more debugging info.

// define variable for HANDLE_CHARSETS out_path so it can be reused later
OUTER_TEXT_SPRINT "HANDLE_CHARSETS_OUT_PATH" EVAL "weidu_external\Lang\%MOD_FOLDER%"

Define variable for HANDLE_CHARSETS out_path so it can be reused later and it will keep it in sync.

tra_path = "%MOD_FOLDER%\lang" // path to translation files, do not add slash at the end of it

Path to translation files, do not add slash at the end of it.

out_path = EVAL "%HANDLE_CHARSETS_OUT_PATH%" // location of converted .tra files

This is very important. WeiDU documentation says:

"Note that if %tra_path and out_path are the same directory, the conversion will naturally happen only once regardless of how many times HANDLE_CHARSETS are invoked, but if the directories are different, the conversion will happen for every invocation (notably, if HANDLE_CHARSETS is invoked among the ALWAYS actions)."

So when WeiDU preforming conversion operation, it overwrite original mod files and it can also change their encoding if the mod was installed on the classic editions of the games. So in order to keep original files intact, we need to provide two different path for `out_path` and `tra_path` parameters. Since WeiDU doesn't have concept of 'temporary work directory' we need to create it manually earlier. Also, if you mod uses AUTRO_TRA, you have to use the "%TRA_LOCATION%" path:

// for the EE games, the variable will point to the untouched .tra files
// for classic games will point to the HANDLE_CHARSETS out_path value where converted files are stored
AUTO_TRA "%TRA_LOCATION%\%s"

The rest of the mod that code covers changing game string and creating of default NPC with initial dialogues are only to quickly test the results and they are not related to the main goal of this guide. Please refer to WeiDU Documentation or ask here.

Edited March 9, 2022 by AL|EN

[Jazira]

Hello Alien,

Thanks for this HANDLE_CHARSETS update.

1 hour ago, AL|EN said:

• DOS/Shell console messages can finally have localized characters displayed properly

That's a very good news !  And tons of setup already translated to proofread.  

1 hour ago, AL|EN said:

• translators no longer send files just to update one single line

I don't really get it, could you explain that one a bit further ? The translator could provide the line via forum, and the modder could copy/past without worry about encoding, that's it ?

Edited January 21, 2021 by Jazira

[AL|EN]

@Jazira Nope, not at all. Text copied from forums/webpages is using UTF8 generally, pasting such encoded text into ASCI files often breaks them. It depends on the editor, codepage and local settings of the modder. Too much thing to handle and too much things that can go wrong here. Murphy's Law.

[Jazira]

1 hour ago, AL|EN said:

translators no longer send files just to update one single line

Then, how do we update a single line without sending a file ?

11 minutes ago, AL|EN said:

pasting such encoded text into ASCI files often breaks them.

I tough the files needs to be in UTF-8 from now on (If the HANDLE_CHARSETS function got updated internally).

[Magus]

So, this is since weidu 247, right? And the old way still works?

And why does the function not handle the "temporary work directory" itself? Is there any reason why'd you want to not have it?

[AL|EN]

@Jazira 

You can update single lines of translation by simply posting it at the forum but only after the mod code and files is updated to use HANDLE_CHARSETS with from_utf8 = 1 parameter. When all source files are using UTF8 (because the mod is EE-only anyway) there is no need to use HANDLE_CHARSETS at all and you also can send translations directly via forum posts etc.

[AL|EN]

4 minutes ago, Magus said:

So, this is since weidu 247, right? And the old way still works?

And why does the function not handle the "temporary work directory" itself? Is there any reason why'd you want to not have it?

Yes, WeiDU 247 is required, old way still works. I strongly believe that there is a reason to update even the old mods which use old way.

You need to ask wisp why providing path to out_path parameter doesn't create it automatically.

[argent77]

Thanks for the hint about the parameter "from_utf8". This is a big improvement over the traditional way of handling translations. No more trouble creating .tra files with the right ANSI encoding.

11 hours ago, AL|EN said:

• DOS/Shell console messages can finally have localized characters displayed properly

This "feature" isn't related in any way to HANDLE_CHARSETS though. It rather appears to be a quirk of the WeiDU binary. The 64-bit Windows executable seems to display UTF-8-encoded setup.tra correctly in the console window, but the 32-bit executable does not. Since most mods ship with 32-bit executables it's still advisable to use only US-ASCII characters in the setup.tra.

Edited January 21, 2021 by argent77
typo

[AL|EN]

It looks to good to be true. It turns out that using out_path which is essential for overcoming HANDLE_CHARSETS quirks is incompatible with AUTO_TRA:

Quote

HANDLE_CHARSETS needs to be used before any text is installed and is compatible with AUTO_TRA and all other methods of loading TRA files, unless the option out_path is used.

I will investigate more, right now you can't use this if you are not wiling to resign from using AUTO_TRA function.

Edited January 28, 2021 by AL|EN

[AL|EN]

Guide was updated to reflect proper and supported usage along with AUTO_TRA. This part of the code was changed:

AUTO_TRA "weidu_external\lang\%MOD_FOLDER%\%s" //use out_path + \%s

You can now use it freely, without limitation.

Edited April 18, 2021 by AL|EN

[AL|EN]

On 1/21/2021 at 11:53 PM, argent77 said:

This "feature" isn't related in any way to HANDLE_CHARSETS though. It rather appears to be a quirk of the WeiDU binary. The 64-bit Windows executable seems to display UTF-8-encoded setup.tra correctly in the console window, but the 32-bit executable does not. Since most mods ship with 32-bit executables it's still advisable to use only US-ASCII characters in the setup.tra.

@argent77 I fond the root cause of this problem, WeiDU 32-bit was compiled using OCalm version without unicode support. More about this here: https://forums.pocketplane.net/index.php?topic=29984.new#new

[Guest Henanigan]

The following code:

ACTION_IF GAME_IS ~eet~ THEN BEGIN
...
  LAF HANDLE_CHARSETS
...
    STR_VAR
      tra_path = ~%MOD_FOLDER%/lang~

...is causing an installation error for me on Linux (set up using a case-insensitive partition; game and some other mods working fine). Changing the tra_path value to an absolute path

tra_path = ~MyModName/lang~

fixes the problem. (Everything seems to work properly in a non-EET environment, i.e. when the GAME_IS above is skipped.) Would this be related to the LANGUAGE issue above?

[AL|EN]

1 hour ago, Guest Henanigan said:

Would this be related to the LANGUAGE issue above?

No because using %MOD_FOLDER% works for HANDLE_CHARSETS function. How about you try to install ull code example:  inside linux https://github.com/ALIENQuake/WeiDU-UTF8 on non -EE and see if it can be installed fine? Also, you can use --debug-assign to see the value of %MOD_FOLDER%

[argent77]

Using variables in string arguments requires the EVAL keyword before the assigned value.

ACTION_IF GAME_IS ~eet~ THEN BEGIN
...
  LAF HANDLE_CHARSETS
...
    STR_VAR
      tra_path = EVAL ~%MOD_FOLDER%/lang~

The keyword "EVAL" can be omitted only if the AUTO_EVAL_STRINGS directive has been defined.

[AL|EN]

@argent77 Yep, good catch. Thanks!