AL|EN Posted January 21 Share Posted January 21 (edited) EDIT: Turns out the code is incompatible with AUTO_TRA. Until future investigation, you can't use this if you are not wiling to resign from using AUTO_TRA function. Introduction: Since introduction of HANDLE_CHARSETS, modders have given the ability to support translation for both classic and enhanced editions of the Infinity Engine games. The function itself was doing it's job: convert file encoding from one to another. But it also had annoyances which various modders was able to overcome via custom code. Now, with the introduction of `out_path` parameter, all of the annoyances are gone. This is an attempt to show how to get the best out of HANDLE_CHARSETS. How it was handled before: translators provide .tra files using their own preferred local encoding HANDLE_CHARSETS function convert those files into UTF8 encoding when mod was installed on enhanced game editions How it is handled now: translators provide .tra files using UTF8 encoding when the mod is installed on classic game editions, the temporary copy of translation files are converted into ASCII encoding so the source files are not touched no modification of original UTF8-files when mod is installed on enhanced game editions Using new way will save you from annoyances and allow for new features: DOS/Shell console messages can finally have localized characters displayed properly modders no longer need to watch-out to not break custom encoding when editing foreign tra files modders can receive translations via forum posts and it can be used instantly, without hustle translators no longer send files just to update one single line translation can be provided by editing files directly via GitHub online interface Prerequisites: required minimum version of WeiDU is 247 WeiDU executable must be 64-bit for DOS/Shell console messages being able to display localized characters properly Full code example: https://github.com/ALIENQuake/WeiDU-UTF8 Spoiler BACKUP "weidu_external\backup\WeiDU-UTF8-Example" // move backup folder outside of the main mod data folder, change only you mod data folder name SUPPORT "forum.com/thread-1" VERSION "0.1.0" AUTO_EVAL_STRINGS ALWAYS OUTER_SPRINT MOD_FOLDER "WeiDU-UTF8-Example" MKDIR "weidu_external\lang\%MOD_FOLDER%" ACTION_DEFINE_ARRAY arrayNoConvert BEGIN setup END ACTION_DEFINE_ARRAY arrayReload BEGIN mod END LAF HANDLE_CHARSETS INT_VAR from_utf8 = 1 infer_charset = 1 verbose = 1 STR_VAR default_language = english tra_path = "%MOD_FOLDER%\lang" out_path = "weidu_external\lang\%MOD_FOLDER%" noconvert_array = arrayNoConvert reload_array = arrayReload END END //Turns out the code is incompatible with AUTO_TRA. Until future investigation, you can't use this if you are not wiling to resign from using AUTO_TRA function. //AUTO_TRA "WeiDU-UTF8-Example\lang\%s" // Don't use %MOD_FOLDER% for LANGUAGE until it will be fixed, http://forums.pocketplane.net/index.php?topic=29521 LANGUAGE "English" "english" "WeiDU-UTF8-Example\lang\english\setup.tra" "WeiDU-UTF8-Example\lang\english\mod.tra" LANGUAGE "Czech" "czech" "WeiDU-UTF8-Example\lang\czech\setup.tra" "WeiDU-UTF8-Example\lang\czech\mod.tra" LANGUAGE "French" "french" "WeiDU-UTF8-Example\lang\french\setup.tra" "WeiDU-UTF8-Example\lang\french\mod.tra" LANGUAGE "German" "german" "WeiDU-UTF8-Example\lang\german\setup.tra" "WeiDU-UTF8-Example\lang\german\mod.tra" LANGUAGE "Italian" "italian" "WeiDU-UTF8-Example\lang\italian\setup.tra" "WeiDU-UTF8-Example\lang\italian\mod.tra" LANGUAGE "Russian" "russian" "WeiDU-UTF8-Example\lang\russian\setup.tra" "WeiDU-UTF8-Example\lang\russian\mod.tra" LANGUAGE "Polish" "polish" "WeiDU-UTF8-Example\lang\polish\setup.tra" "WeiDU-UTF8-Example\lang\polish\mod.tra" LANGUAGE "Spanish" "spanish" "WeiDU-UTF8-Example\lang\spanish\setup.tra" "WeiDU-UTF8-Example\lang\spanish\mod.tra" BEGIN @1001 ACTION_IF GAME_IS "bg1 totsc bgee" BEGIN STRING_SET 15594 @10001 END ACTION_IF GAME_IS "bgt" BEGIN STRING_SET 11676 @10001 END ACTION_IF GAME_IS "bg2ee tob" BEGIN STRING_SET 70742 @10001 END ACTION_IF GAME_IS "eet" BEGIN STRING_SET 215594 @10001 // +200000 END Code explanation: BACKUP "weidu_external\backup\WeiDU-UTF8-Example" // move backup folder outside of the main mod data folder Move backup folder outside of the main mod data folder. The %MOD_FOLDER% is read from this value but we will change it later to the correct mod data directory. ACTION_DEFINE_ARRAY arrayNoConvert BEGIN setup END ACTION_DEFINE_ARRAY arrayReload BEGIN mod END Define two arrays for `noconvert_array` and `reload_array` parameters of `HANDLE_CHARSETS` function because those parameters require array. You cannot provide the filenames directly. The .tra file extension is optional. LAF HANDLE_CHARSETS INT_VAR from_utf8 = 1 //the source tra files of all languages are UTF8 infer_charset = 1 //set the correct encoding for classic games based on the detected game and associated encoding verbose = 1 //display more debugging info Parameter from_utf8 = 1will tell WeiDU that the source tra files of all languages are UTF8. The infer_charset = 1 will set the correct encoding for classic games based on the detected game and associated encoding. The verbose = 1 will display more debugging info. tra_path = "%MOD_FOLDER%\lang" // path to translation files, do not add slash at the end of it Path to translation files, do not add slash at the end of it. out_path = "weidu_external\lang\%MOD_FOLDER%" // path to temporary work directory, do not add slash at the end of it This is very important. WeiDU documentation says: "Note that if %tra_path and out_path are the same directory, the conversion will naturally happen only once regardless of how many times HANDLE_CHARSETS are invoked, but if the directories are different, the conversion will happen for every invocation (notably, if HANDLE_CHARSETS is invoked among the ALWAYS actions)." So when WeiDU preforming conversion operation, it overwrite original mod files and it can also change their encoding if the mod was installed on the classic editions of the games. So in order to keep original files intact, we need to provide two different path for `out_path` and `tra_path` parameters. Since WeiDU doesn't have concept of 'temporary work directory' we need to create it manually earlier. OUTER_SPRINT MOD_FOLDER "WeiDU-UTF8-Example" MKDIR "weidu_external\lang\%MOD_FOLDER%" Create 'temporary work directory'. Additionally, we are make sure that %MOD_FOLDER% variable have proper value for mod data directory. // Don't use %MOD_FOLDER% for LANGUAGE until it will be fixed, http://forums.pocketplane.net/index.php?topic=29521 LANGUAGE "English" "english" "WeiDU-UTF8-Example\lang\english\setup.tra" LANGUAGE "Czech" "czech" "WeiDU-UTF8-Example\lang\czech\setup.tra" LANGUAGE "French" "french" "WeiDU-UTF8-Example\lang\french\setup.tra" LANGUAGE "German" "german" "WeiDU-UTF8-Example\lang\german\setup.tra" LANGUAGE "Italian" "italian" "WeiDU-UTF8-Example\lang\italian\setup.tra" LANGUAGE "Russian" "russian" "WeiDU-UTF8-Example\lang\russian\setup.tra" LANGUAGE "Polish" "polish" "WeiDU-UTF8-Example\lang\polish\setup.tra" LANGUAGE "Spanish" "spanish" "WeiDU-UTF8-Example\lang\spanish\setup.tra" Don't use %MOD_FOLDER% for LANGUAGE until it will be fixed, http://forums.pocketplane.net/index.php?topic=29521 The rest of the mod code should be self-explanatory. If not, please refer to WeiDU Documentation or ask here. Edited January 28 by AL|EN Quote Link to post
Jazira Posted January 21 Share Posted January 21 (edited) Hello Alien, Thanks for this HANDLE_CHARSETS update. 1 hour ago, AL|EN said: DOS/Shell console messages can finally have localized characters displayed properly That's a very good news ! And tons of setup already translated to proofread. 1 hour ago, AL|EN said: translators no longer send files just to update one single line I don't really get it, could you explain that one a bit further ? The translator could provide the line via forum, and the modder could copy/past without worry about encoding, that's it ? Edited January 21 by Jazira Quote Link to post
AL|EN Posted January 21 Author Share Posted January 21 @Jazira Nope, not at all. Text copied from forums/webpages is using UTF8 generally, pasting such encoded text into ASCI files often breaks them. It depends on the editor, codepage and local settings of the modder. Too much thing to handle and too much things that can go wrong here. Murphy's Law. Quote Link to post
Jazira Posted January 21 Share Posted January 21 1 hour ago, AL|EN said: translators no longer send files just to update one single line Then, how do we update a single line without sending a file ? 11 minutes ago, AL|EN said: pasting such encoded text into ASCI files often breaks them. I tough the files needs to be in UTF-8 from now on (If the HANDLE_CHARSETS function got updated internally). Quote Link to post
Magus Posted January 21 Share Posted January 21 So, this is since weidu 247, right? And the old way still works? And why does the function not handle the "temporary work directory" itself? Is there any reason why'd you want to not have it? Quote Link to post
AL|EN Posted January 21 Author Share Posted January 21 @Jazira You can update single lines of translation by simply posting it at the forum but only after the mod code and files is updated to use HANDLE_CHARSETS with from_utf8 = 1 parameter. When all source files are using UTF8 (because the mod is EE-only anyway) there is no need to use HANDLE_CHARSETS at all and you also can send translations directly via forum posts etc. Quote Link to post
AL|EN Posted January 21 Author Share Posted January 21 4 minutes ago, Magus said: So, this is since weidu 247, right? And the old way still works? And why does the function not handle the "temporary work directory" itself? Is there any reason why'd you want to not have it? Yes, WeiDU 247 is required, old way still works. I strongly believe that there is a reason to update even the old mods which use old way. You need to ask wisp why providing path to out_path parameter doesn't create it automatically. Quote Link to post
argent77 Posted January 21 Share Posted January 21 (edited) Thanks for the hint about the parameter "from_utf8". This is a big improvement over the traditional way of handling translations. No more trouble creating .tra files with the right ANSI encoding. 11 hours ago, AL|EN said: DOS/Shell console messages can finally have localized characters displayed properly This "feature" isn't related in any way to HANDLE_CHARSETS though. It rather appears to be a quirk of the WeiDU binary. The 64-bit Windows executable seems to display UTF-8-encoded setup.tra correctly in the console window, but the 32-bit executable does not. Since most mods ship with 32-bit executables it's still advisable to use only US-ASCII characters in the setup.tra. Edited January 21 by argent77 typo Quote Link to post
AL|EN Posted January 28 Author Share Posted January 28 (edited) It looks to good to be true. It turns out that using out_path which is essential for overcoming HANDLE_CHARSETS quirks is incompatible with AUTO_TRA: Quote HANDLE_CHARSETS needs to be used before any text is installed and is compatible with AUTO_TRA and all other methods of loading TRA files, unless the option out_path is used. I will investigate more, right now you can't use this if you are not wiling to resign from using AUTO_TRA function. Edited January 28 by AL|EN Quote Link to post
Recommended Posts