[tutorial] Slightly streamlined handling of tra files via HANDLE_CHARSETS

[DavidW]

This is an attempt to make handling languages on EE and non-EE games a bit smoother, building on work by @AL|EN (here) and myself (here).

You need to put the attached file into the 'lib' subfolder of your mod folder (i.e. mymod/lib).

Then, providing

(1) you keep your .tra files in 'mymod/tra' or 'mymod/lang'

(2) the names of your components, and similar strings used by WEIDU while installing, live in setup.tra

(3) you don't want to automatically load any tra files except setup.tra (you're going to load them all using LOAD_TRA and WITH_TRA, and/or use USING and AUTO_TRA)

(4) your .tra files are encoded in UTF-8

(5) you wrote the mod in English originally (so all tra files are definitely there in English)

all you need to do is

BACKUP ~weidu_external/backup/mymod~
AUTHOR "DavidW [insert link to forum page]"
VERSION ~[whatever]~

ALWAYS
	INCLUDE "%MOD_FOLDER%/lib/charset_wrapper.tph"
	LAF charset_wrapper RET out_path END
END 
AUTO_TRA "%out_path%/%s"

LANGUAGE "English" english "test/lang/english/setup.tra"
LANGUAGE "Old English" oldenglish "test/lang/english/setup.tra" "test/lang/oldenglish/setup.tra"

In case any of (1)-(5) aren't true, you can adjust for them by inputs to the charset_wrapper function, like this:

BACKUP ~weidu_external/backup/mymod~
AUTHOR "DavidW [insert link to forum page]"
VERSION ~[whatever]~

ALWAYS
	INCLUDE "%MOD_FOLDER%/lib/charset_wrapper.tph"
	LAF charset_wrapper 
		INT_VAR 
			from_utf8=1 // set this to 0 if your tra files are encoded for non-EE games and need to be converted to utf-8
			overwrite=0 // set this to 1 if you want the function to redo the conversion every time you install a new
				// component. (I recommend doing this while developing so that changes you make to the .tra files
				// get caught immediately.)
			verbose=0 // set this to 1 to get more feedback from HANDLE_CHARSETS
		STR_VAR 
			tra_path="" // set this to the folder containing your tra files, if it's not 'tra' or 'lang'. It doesn't matter
					// whether or not you include the mod folder root, i.e. 'mymod/mytra' and 'mytra' get treated the same
			setup="setup" // set this to the name of the tra file with your component names in, if it's not setup.tra
			load="" // set this to a list of tra files you want to always be loaded, separated by spaces. (You can include 
				// the '.tra' or not as you like, it doesn't matter)
			default_language="english" // set this to the language you wrote the mod in
RET out_path END
END 
AUTO_TRA "%out_path%/%s"

LANGUAGE "English" english "test/lang/english/setup.tra"
LANGUAGE "Old English" oldenglish "test/lang/english/setup.tra" "test/lang/oldenglish/setup.tra"

Functionally, the main differences with earlier versions are:

• we copy tra files over to weidu_external/lang/%LANGUAGE% whether or not iconv is run. The main point of doing so is to make sure all the tra files are present, by using default-language versions if they're not present on the desired language. HANDLE_CHARSETS does this automatically, but it is good to have it happen even if we don't need to convert to/from UTF-8.
• the whole thing only runs once (not once per component), which saves time.
• Hopefully the interface is a bit easier for novices.

Here's the actual function:

Spoiler

DEFINE_ACTION_FUNCTION charset_wrapper
	INT_VAR from_utf8=1 // set to 0 if your tra files aren't in UTF8
			overwrite=0	 // set to 1 if you want to regenerate the converted files every run
			verbose=0	// set to 1 to get more feedback from HANDLE_CHARSETS
	STR_VAR	tra_path="" // set to where your tra files are (with or without '%MOD_FOLDER%'
						// if they're not in %MOD_FOLDER%/tra or %MOD_FOLDER%/lang
			setup_tra="setup"	// set to whatever you're keeping your WEIDU installation strings in
			load=""	// set to a space-separated list of any tra files you want loaded 
			default_language="english" // set to whatever language you wrote the mod in
	RET out_path
BEGIN
	// set out path
	OUTER_SPRINT out_path "weidu_external/lang/%MOD_FOLDER%"
	// find the tra location
	ACTION_IF "%tra_path%" STR_EQ "" BEGIN
		// try to guess 'tra' or 'lang'
		ACTION_IF DIRECTORY_EXISTS "%MOD_FOLDER%/lang" && DIRECTORY_EXISTS "%MOD_FOLDER%/tra" BEGIN
			FAIL "charset_wrapper error: %MOD_FOLDER% contains both tra and lang subfolders(!) You need to specify which one you use for your tra files."
		END ELSE
		ACTION_IF DIRECTORY_EXISTS "%MOD_FOLDER%/lang" BEGIN
			OUTER_SPRINT tra_path "%MOD_FOLDER%/lang"
		END ELSE
		ACTION_IF DIRECTORY_EXISTS "%MOD_FOLDER%/tra" BEGIN
			OUTER_SPRINT tra_path "%MOD_FOLDER%/tra"
		END ELSE BEGIN
			FAIL "charset_wrapper error: you didn't specify tra_path and it's not 'tra' or 'lang'."
		END
	END ELSE 
	// add '%MOD_FOLDER%' if it's not there already
	ACTION_IF !(INDEX ("%MOD_FOLDER%" "%tra_path%")=0) BEGIN
		OUTER_SPRINT tra_path "%MOD_FOLDER%/%tra_path%"
	END
	ACTION_IF overwrite BEGIN 
		OUTER_SET proceed=1 // always proceed if overwrite=1
	END ELSE BEGIN
		OUTER_SET proceed=1
		ACTION_BASH_FOR "%out_path%/%LANGUAGE%" ".*\.tra$" BEGIN
			OUTER_SET proceed=0 // proceed if out_path is empty
		END
	END
	ACTION_IF proceed BEGIN
		// set the no-copy array
		ACTION_CLEAR_ARRAY noconvert_array
		OUTER_SPRINT $noconvert_array(0) "%setup_tra%"
		// set the reload array
		ACTION_CLEAR_ARRAY reload_array
		OUTER_PATCH "%load% " BEGIN
			ind=0
			REPLACE_EVALUATE "\([^ ]+\) " BEGIN
				SPRINT $reload_array("%ind%") "%MATCH1%"
				++ind
			END
			""
		END
		// are we on enhanced edition?
		// later versions of EE have bgee.lua. Very early versions of BGEE have monkfist.2da
		OUTER_SET enhanced_edition=( FILE_EXISTS_IN_GAME bgee.lua || FILE_EXISTS_IN_GAME monkfist.2da )
		// are we converting?
		OUTER_SET convert=!(enhanced_edition=from_utf8)
		ACTION_IF convert && !FILE_EXISTS "%tra_path%/iconv/iconv.exe" BEGIN
			OUTER_SET convert=0
			WARN "charset_wrapper warning: can't find iconv.exe"
		END
		ACTION_IF convert BEGIN
			// run HANDLE_CHARSETS
			LAF HANDLE_CHARSETS
				INT_VAR from_utf8=from_utf8
						infer_charsets=1
						verbose=verbose
				STR_VAR tra_path = EVAL "%tra_path%" // I hate not being able to assume AUTO_EVAL_STRINGS
						out_path = EVAL "%out_path%"
						noconvert_array = noconvert_array
						reload_array = reload_array
						default_language = EVAL "%default_language%"
			END
		END ELSE BEGIN
			// do our own copies
			MKDIR "%out_path%"
			ACTION_BASH_FOR "%tra_path%/%default_language%" ".*\.tra$" BEGIN
				ACTION_IF !"%BASH_FOR_RES%" STR_EQ "%setup" BEGIN // don't bother copying this
					ACTION_IF FILE_EXISTS "%tra_path%/%LANGUAGE%/%BASH_FOR_FILE%" BEGIN
						OUTER_SPRINT tra "%tra_path%/%LANGUAGE%/%BASH_FOR_FILE%"
					END ELSE BEGIN
						OUTER_SPRINT tra "%tra_path%/%default_language%/%BASH_FOR_FILE%"
					END
					COPY "%tra%" "%out_path%/%LANGUAGE%"
				END
			END	
			// do our own reloads
			ACTION_PHP_EACH reload_array AS int=>tra BEGIN
				ACTION_IF "%tra%" STRING_MATCHES_REGEXP ".+\.tra$" = 0 BEGIN
					LOAD_TRA "%out_path%/%LANGUAGE%/%tra%"				
				END ELSE BEGIN
					LOAD_TRA "%out_path%/%LANGUAGE%/%tra%.tra"
				END
			END
		END
	END
END

 

Feedback (and bug fixes!) welcomed.

 

charset_wrapper.tph

[AL|EN]

Congratulations on the release! What do you think about the inclusion of this into weidu itself, for eg: HANDLE_CHARSETS_EX?

[DavidW]

I’m pretty relaxed about whether something is formally included in WEIDU or not - it’s easy enough to INCLUDE a file. (And I don’t anticipate a new release of WEIDU any time soon.)

[jastey]

On 10/10/2023 at 12:14 AM, DavidW said:

(3) you don't want to automatically load any tra files except setup.tra (you're going to load them all using LOAD_TRA and WITH_TRA, and/or use USING and AUTO_TRA)

Is this a prerequisite for your function to work or could this be expanded upon? - It's common for mods to have all install strings in setup.tra, but ingame strings in another file (e.g. game.tra), then there is the EE specific game_ee.tra which needs to be reloaded (in case of ANSI -> utf8 conversion), and journal entries are sometimes in a journal.tra, all loaded in the LANGUAGE definition because the strings are either used in the tp2 or over several d/baf files (for journal entries, because they need to match exactly for EraseJournalEntry to work).

[DavidW]

It’s fine, you just need to call them out explicitly (same as HANDLE_CHARSETS, basically). In your example, just add (as a STR_VAR) 

load="game game_ee journal"

 

[jastey]

Ah, I see. Thanks!

[DavidW]

Test it before using in something live, though! I’m pretty sure it’s ok, but not 100%

[DavidW]

Caught a problem with the program if the directory hadn't already been created (new version now uploaded).