translation [Tutorial] How to properly handle mod translation by using HANDLE_CHARSETS function

[AL|EN]

EDIT: Turns out the code is incompatible with AUTO_TRA. Until future investigation, you can't use this if you are not wiling to resign from using AUTO_TRA function.

Introduction:

Since introduction of HANDLE_CHARSETS, modders have given the ability to support translation for both classic and enhanced editions of the Infinity Engine games.
The function itself was doing it's job: convert file encoding from one to another. But it also had annoyances which various modders was able to overcome via custom code.
Now, with the introduction of `out_path` parameter, all of the annoyances are gone. This is an attempt to show how to get the best out of HANDLE_CHARSETS.

How it was handled before:

1. translators provide .tra files using their own preferred local encoding
2. HANDLE_CHARSETS function convert those files into UTF8 encoding when mod was installed on enhanced game editions

How it is handled now:

1. translators provide .tra files using UTF8 encoding
2. when the mod is installed on classic game editions, the temporary copy of translation files are converted into ASCII encoding so the source files are not touched
3. no modification of original UTF8-files when mod is installed on enhanced game editions

Using new way will save you from annoyances and allow for new features:

• DOS/Shell console messages can finally have localized characters displayed properly
• modders no longer need to watch-out to not break custom encoding when editing foreign tra files
• modders can receive translations via forum posts and it can be used instantly, without hustle 
• translators no longer send files just to update one single line
• translation can be provided by editing files directly via GitHub online interface

Prerequisites:

• required minimum version of WeiDU is 247
• WeiDU executable must be 64-bit for DOS/Shell console messages being able to display localized characters properly

Full code example: https://github.com/ALIENQuake/WeiDU-UTF8

Spoiler

 BACKUP "weidu_external\backup\WeiDU-UTF8-Example" // move backup folder outside of the main mod data folder, change only you mod data folder name
SUPPORT "forum.com/thread-1"
VERSION "0.1.0"

AUTO_EVAL_STRINGS

ALWAYS
    OUTER_SPRINT MOD_FOLDER "WeiDU-UTF8-Example"
    MKDIR "weidu_external\lang\%MOD_FOLDER%"

    ACTION_DEFINE_ARRAY arrayNoConvert BEGIN setup END
    ACTION_DEFINE_ARRAY arrayReload BEGIN mod END

    LAF HANDLE_CHARSETS
        INT_VAR
            from_utf8 = 1
            infer_charset = 1
            verbose = 1
        STR_VAR
            default_language = english
            tra_path = "%MOD_FOLDER%\lang"
            out_path = "weidu_external\lang\%MOD_FOLDER%"
            noconvert_array = arrayNoConvert
            reload_array = arrayReload
    END
END

//Turns out the code is incompatible with AUTO_TRA. Until future investigation, you can't use this if you are not wiling to resign from using AUTO_TRA function.
//AUTO_TRA "WeiDU-UTF8-Example\lang\%s"

// Don't use %MOD_FOLDER% for LANGUAGE until it will be fixed, http://forums.pocketplane.net/index.php?topic=29521
LANGUAGE "English" "english" "WeiDU-UTF8-Example\lang\english\setup.tra" "WeiDU-UTF8-Example\lang\english\mod.tra"
LANGUAGE "Czech" "czech" "WeiDU-UTF8-Example\lang\czech\setup.tra" "WeiDU-UTF8-Example\lang\czech\mod.tra"
LANGUAGE "French" "french" "WeiDU-UTF8-Example\lang\french\setup.tra" "WeiDU-UTF8-Example\lang\french\mod.tra"
LANGUAGE "German" "german" "WeiDU-UTF8-Example\lang\german\setup.tra" "WeiDU-UTF8-Example\lang\german\mod.tra"
LANGUAGE "Italian" "italian" "WeiDU-UTF8-Example\lang\italian\setup.tra" "WeiDU-UTF8-Example\lang\italian\mod.tra"
LANGUAGE "Russian" "russian" "WeiDU-UTF8-Example\lang\russian\setup.tra" "WeiDU-UTF8-Example\lang\russian\mod.tra"
LANGUAGE "Polish" "polish" "WeiDU-UTF8-Example\lang\polish\setup.tra" "WeiDU-UTF8-Example\lang\polish\mod.tra"
LANGUAGE "Spanish" "spanish" "WeiDU-UTF8-Example\lang\spanish\setup.tra" "WeiDU-UTF8-Example\lang\spanish\mod.tra"
BEGIN @1001

ACTION_IF GAME_IS "bg1 totsc bgee" BEGIN
    STRING_SET 15594 @10001
END

ACTION_IF GAME_IS "bgt" BEGIN
    STRING_SET 11676 @10001
END

ACTION_IF GAME_IS "bg2ee tob" BEGIN
    STRING_SET 70742 @10001
END

ACTION_IF GAME_IS "eet" BEGIN
    STRING_SET 215594 @10001 // +200000
END

 

Code explanation:

 BACKUP "weidu_external\backup\WeiDU-UTF8-Example" // move backup folder outside of the main mod data folder

Move backup folder outside of the main mod data folder. The %MOD_FOLDER% is read from this value but we will change it later to the correct mod data directory.

ACTION_DEFINE_ARRAY arrayNoConvert BEGIN setup END
ACTION_DEFINE_ARRAY arrayReload BEGIN mod END

Define two arrays for `noconvert_array` and `reload_array` parameters of `HANDLE_CHARSETS` function because those parameters require array. You cannot provide the filenames directly. The .tra file extension is optional.

LAF HANDLE_CHARSETS
    INT_VAR
        from_utf8 = 1 //the source tra files of all languages are UTF8
        infer_charset = 1 //set the correct encoding for classic games based on the detected game and associated encoding
        verbose = 1 //display more debugging  info

Parameter from_utf8 = 1will tell WeiDU that the source tra files of all languages are UTF8. The infer_charset = 1 will set the correct encoding for classic games based on the detected game and associated encoding. The verbose = 1 will display more debugging info.

        tra_path = "%MOD_FOLDER%\lang" // path to translation files, do not add slash at the end of it

Path to translation files, do not add slash at the end of it.

        out_path = "weidu_external\lang\%MOD_FOLDER%" // path to temporary work directory, do not add slash at the end of it

This is very important. WeiDU documentation says:

"Note that if %tra_path and out_path are the same directory, the conversion will naturally happen only once regardless of how many times HANDLE_CHARSETS are invoked, but if the directories are different, the conversion will happen for every invocation (notably, if HANDLE_CHARSETS is invoked among the ALWAYS actions)."

So when WeiDU preforming conversion operation, it overwrite original mod files and it can also change their encoding if the mod was installed on the classic editions of the games. So in order to keep original files intact, we need to provide two different path for `out_path` and `tra_path` parameters. Since WeiDU doesn't have concept of 'temporary work directory' we need to create it manually earlier.

OUTER_SPRINT MOD_FOLDER "WeiDU-UTF8-Example"
MKDIR "weidu_external\lang\%MOD_FOLDER%"

Create 'temporary work directory'. Additionally, we are make sure that %MOD_FOLDER% variable have proper value for mod data directory.

// Don't use %MOD_FOLDER% for LANGUAGE until it will be fixed, http://forums.pocketplane.net/index.php?topic=29521
LANGUAGE "English" "english" "WeiDU-UTF8-Example\lang\english\setup.tra"
LANGUAGE "Czech" "czech" "WeiDU-UTF8-Example\lang\czech\setup.tra"
LANGUAGE "French" "french" "WeiDU-UTF8-Example\lang\french\setup.tra"
LANGUAGE "German" "german" "WeiDU-UTF8-Example\lang\german\setup.tra"
LANGUAGE "Italian" "italian" "WeiDU-UTF8-Example\lang\italian\setup.tra"
LANGUAGE "Russian" "russian" "WeiDU-UTF8-Example\lang\russian\setup.tra"
LANGUAGE "Polish" "polish" "WeiDU-UTF8-Example\lang\polish\setup.tra"
LANGUAGE "Spanish" "spanish" "WeiDU-UTF8-Example\lang\spanish\setup.tra"

Don't use %MOD_FOLDER% for LANGUAGE until it will be fixed, http://forums.pocketplane.net/index.php?topic=29521

The rest of the mod code should be self-explanatory. If not, please refer to WeiDU Documentation or ask here.

Edited January 28 by AL|EN

[Jazira]

Hello Alien,

Thanks for this HANDLE_CHARSETS update.

1 hour ago, AL|EN said:

• DOS/Shell console messages can finally have localized characters displayed properly

That's a very good news !  And tons of setup already translated to proofread.  

1 hour ago, AL|EN said:

• translators no longer send files just to update one single line

I don't really get it, could you explain that one a bit further ? The translator could provide the line via forum, and the modder could copy/past without worry about encoding, that's it ?

Edited January 21 by Jazira

[AL|EN]

@Jazira Nope, not at all. Text copied from forums/webpages is using UTF8 generally, pasting such encoded text into ASCI files often breaks them. It depends on the editor, codepage and local settings of the modder. Too much thing to handle and too much things that can go wrong here. Murphy's Law.

[Jazira]

1 hour ago, AL|EN said:

translators no longer send files just to update one single line

Then, how do we update a single line without sending a file ?

11 minutes ago, AL|EN said:

pasting such encoded text into ASCI files often breaks them.

I tough the files needs to be in UTF-8 from now on (If the HANDLE_CHARSETS function got updated internally).

[Magus]

So, this is since weidu 247, right? And the old way still works?

And why does the function not handle the "temporary work directory" itself? Is there any reason why'd you want to not have it?

[AL|EN]

@Jazira 

You can update single lines of translation by simply posting it at the forum but only after the mod code and files is updated to use HANDLE_CHARSETS with from_utf8 = 1 parameter. When all source files are using UTF8 (because the mod is EE-only anyway) there is no need to use HANDLE_CHARSETS at all and you also can send translations directly via forum posts etc.

[AL|EN]

4 minutes ago, Magus said:

So, this is since weidu 247, right? And the old way still works?

And why does the function not handle the "temporary work directory" itself? Is there any reason why'd you want to not have it?

Yes, WeiDU 247 is required, old way still works. I strongly believe that there is a reason to update even the old mods which use old way.

You need to ask wisp why providing path to out_path parameter doesn't create it automatically.

[argent77]

Thanks for the hint about the parameter "from_utf8". This is a big improvement over the traditional way of handling translations. No more trouble creating .tra files with the right ANSI encoding.

11 hours ago, AL|EN said:

• DOS/Shell console messages can finally have localized characters displayed properly

This "feature" isn't related in any way to HANDLE_CHARSETS though. It rather appears to be a quirk of the WeiDU binary. The 64-bit Windows executable seems to display UTF-8-encoded setup.tra correctly in the console window, but the 32-bit executable does not. Since most mods ship with 32-bit executables it's still advisable to use only US-ASCII characters in the setup.tra.

Edited January 21 by argent77
typo

[AL|EN]

It looks to good to be true. It turns out that using out_path which is essential for overcoming HANDLE_CHARSETS quirks is incompatible with AUTO_TRA:

Quote

HANDLE_CHARSETS needs to be used before any text is installed and is compatible with AUTO_TRA and all other methods of loading TRA files, unless the option out_path is used.

I will investigate more, right now you can't use this if you are not wiling to resign from using AUTO_TRA function.

Edited January 28 by AL|EN