SFO alpha

[DavidW]

I've put an alpha version of my "Stratagems Functional Overlay" (SFO - better names welcomed!) function library in the download centre, mostly to make its iterative resource checker available to a couple of people who asked about it. Most of its functionality is currently undocumented and it's probably still got some bugs - use at own risk. (I'm hoping to make a properly-documented beta version available some time in the next few months.

 

Download link here. I'll put instructions for a couple of bits and pieces below.

[DavidW]

Installing:

 

1) Put the SFO folder in your mod directory.

2) Put AUTO_EVAL_STRINGS into your mod preamble.

3) Add the following to the ALWAYS block of your mod:

 

INCLUDE mymod/sfo/install_sfo.tpa~

(where "mymod" is the directory of your mod, obviously)

 

4) If your mod is not using standard naming conventions (i.e. if WEIDU's TP2_BASE_NAME doesn't correctly pick out the name of your mod directory) you need to add "

OUTER_SPRINT scsroot mymod" to the ALWAYS block.

 

5) Optionally, add "OUTER_SPRINT workspace [some directory]" to the ALWAYS block. (This determines the default workspace where SFO makes things and saves logs. It defaults to stratagems_external/workspace.)

 

6) Optionally, add "OUTER_SET read_resources=0" to the ALWAYS block. By default SFO reads in a bunch of information when first run; if you don't want that information (see next entry) you'll save a small amount of time by not reading it in. (The most user-visible such bit of information is readable names for spells: for any spell in SPELL.IDS, a variable with the same name as the ids name is set to the spell resref and a variable with that name followed by "_SCROLL" is set to the resref of a scroll of that spell(so "WIZARD_FIREBALL" is set to spwi304 and "WIZARD_FIREBALL_SCROLL" is set to "scrl5z"). Entries are also set for the avenger wizard spells, using names like CLERIC_CHROMATIC_ORB).

[DavidW]

Using the iterative resource checker:

 

This is a function that takes as input a collection of game resources, runs whatever operation you like on them, then collects all the game resources that those resources in turn use, and iterates the process. Possible uses include (i) looking for missing items; (ii) finding files that aren't actually used in-game.

 

The formal syntax is as follows (nb this is slightly different from the version I previewed last month):

 

- the function itself is iter_resource, which is an action function

- it takes one integer argument, include_spells (default value 0), and three string arguments: to_check (default value ""), ignore (default value "") and func (no default value; it'll crash if nothing is specified), and has one output, "output".

- to_check is the list of files to start off with, in full and separated by spaces e.g. "spwi501.spl misc01.itm".

- to_ignore is a possibly-empty list of file extensions separated by spaces, e.g. "are bcs". iter_func doesn't iterate through any files with these extensions unless they are on the original list to check. So for instance, if you want to check which files are missing from a given area, put ignore="are" to prevent iter_func iterating through all the areas connected to it, and all the ones connected to that area...

- if include_spells is set to 1, iter_func adds to the check list all spells referenced in spell.ids or in any clab or lua file.

- func is a function, which is executed on every filename returned by iter_func.

 

The format of func, in turn, is as follows. It must be an action function, and take as input four string variables, "output", "resref", "ext", and "parent", and return "output". Of these, "resref" and "ext" are the main and extension part of the file being checked, "parent" is the file which uses it, and "output" is initially "" and is passed through the function on each iteration. (Each pair of child file and parent file has func applied to it only once.)

 

That might sound a bit forbiddingly abstract. Here's a simple example:

DEFINE_ACTION_FUNCTION simple_func
  STR_VAR output=""
	   resref=""
	   ext=""
	   parent=""
  RET output
BEGIN
OUTER_SPRINT output "%output% %resref%.%ext% (used by %parent%)"
END

 

If you then do

LAF iter_func STR_VAR to_check="spwi501.spl" func=simple_func RET output END
PRINT "%output%"

you'll get a list of every resource required by the "animate dead" spell.

 

For a more useful example, consider

DEFINE_ACTION_FUNCTION iterfunc_missing_file
  STR_VAR resref=""
	   ext=""
	   parent=""
	   output=""
  RET	 output
BEGIN
  ACTION_IF !FILE_EXISTS_IN_GAME "%resref%.%ext%" BEGIN
  OUTER_SET report=1
  ACTION_MATCH "%resref%" WITH
  "rndtre.*" "rndmag.*"
  BEGIN
	OUTER_SET report=0
  END
  DEFAULT
  END
  ACTION_IF "%ext%" STRING_EQUAL bcs BEGIN
	 ACTION_MATCH "%resref%.are" WITH
	 "%parent%" BEGIN
			 OUTER_SET report=0
	 END
	 DEFAULT
	 END
  END
  ACTION_IF report BEGIN
	 LAF push STR_VAR tail="%output%" head="%resref%.%ext% (used by %parent%)" RET output=list END
  END
  END
END

This collects a list of files that are missing. If you do

LAF iter_func INT_VAR include_spells=1 STR_VAR func=iterfunc_missing_file to_check="ar2600.are worldmap.wmp" RET output END
LAF write_list STR_VAR file="missing.txt" list="%output%" END

then the file "missing.txt", located in the "workspace" directory, will contain every resource that is missing from the main part of BGEE or from the BGEE spell system.

 

The code is not especially optimised for speed. If you do a wideranging iter_func, go and have a coffee or something while it's running.

[DavidW]

I should admit that if I try to run this on worldmap.wmp in BG2 I get an out-of-memory error in WEIDU. (It handles ToB, BG1, IWD fine, but I think the sheer number of resources confuses it.) If anyone wants to use this for SoA, probably just doing areas a few dozen at a time and setting ignore=are is the best way - it's easy enough to tell which areas are and aren't in use, getting them off the worldmap is just a convenience. Here's a function listing all areas of ARwxyz format between two integers from and to:

 

DEFINE_ACTION_FUNCTION return_area_list 
   INT_VAR from=0
           to=0
   STR_VAR list="" 
   RET list
BEGIN
   ACTION_IF from<to BEGIN
       ACTION_IF from<10 BEGIN 
           OUTER_SPRINT to_check "ar000%from%"
       END ELSE
       ACTION_IF from<100 BEGIN
           OUTER_SPRINT to_check "ar00%from%"
       END ELSE
       ACTION_IF from<1000 BEGIN
           OUTER_SPRINT to_check "ar0%from%"
       END ELSE BEGIN
           OUTER_SPRINT to_check "ar%from%"
       END 
       ACTION_IF FILE_EXISTS_IN_GAME "%to_check%.are" BEGIN
          OUTER_SPRINT list "%list% %to_check%.are"
       END
       LAF return_area_list INT_VAR from=from+1 to STR_VAR list RET list END
   END
END

I find that

LAF return_area_list INT_VAR from=0 to=1000 RET to_check=list END
LAF iter_resource INT_VAR STR_VAR func=iterfunc_missing_file to_check ignore=are RET output END
LAF write_list STR_VAR list="%output%" file=missing_game_resources.txt  END

works fine, for instance. (though it also shows that the current version doesn't handle pseudocreatures like RDGOB correctly - c'est la vie.)

[Miloch]

Let's say we have a scenario where we want to determine an "unused" list of all files of a certain type (e.g. TIS, MOS, WAV etc.) in that they are not referenced in any other relevant files. Would this be able to help with that and if so, could you produce a usage example?

[DavidW]

Yep. (That's what I originally built it for, actually, in the context of IWD-in-BG2.)

 

(1) Get a list of all files of that type that *are* used.

(2) Go through all files of the type and check them off against that list.

(3) See what's left.

 

Here's a quick attempt at code that would do it for BGEE (I'm typing from work so can't easily test it):

 

DEFINE_ACTION_FUNCTION list_if_wav 
  STR_VAR output="" resref="" ext="" parent="" 
  RET output
BEGIN
  ACTION_IF "%ext%" STRING_EQUAL_CASE wav BEGIN
     OUTER_SPRINT output "%output% %resref%"
  END
END

LAF iter_func STR_VAR to_check="worldmap.wmp ar2600.are" func=list_if_wav RET list=output END // collects a list of all wav resrefs

OUTER_WHILE "%output%" STRING_COMPARE "" BEGIN
   LAF return_first_entry STR_VAR list RET entry list END
   ACTION_TO_LOWER entry 
   OUTER_SPRINT $this_wav_used("%entry%") ""
END

OUTER_SPRINT not_used ""

COPY_EXISTING_REGEXP ".*\.wav" "override"
    SPRINT this_wav "%SOURCE_RES%"
    TO_LOWER this_wav 
    PATCH_IF !VARIABLE_IS_SET $this_wav_used("%this_wav%") BEGIN
        OUTER_SPRINT not_used "%not_used% %this_wav%"
    END
BUT_ONLY

LAF write_list STR_VAR file="wav_not_used.txt" list="%not_used%" END

 

For BG2, the out-of-memory problem I mentioned above means you'd probably have to make the list of wav files used in two or three goes. And you shouldn't rely on it blindly - it doesn't collect wavs used in dialog strings, nor those that are hardcoded.

[DavidW]

On BG2, using the return_area_list function I defined earlier, you could replace the single call to iter_func with

 

LAF return_area_list INT_VAR from=0 to=1000 RET to_check=list END
LAF iter_resource INT_VAR STR_VAR func=list_if_wav to_check ignore=are RET list1=output END
LAF return_area_list INT_VAR from=1001 to=2000 RET to_check=list END
LAF iter_resource INT_VAR STR_VAR func=list_if_wav to_check ignore=are RET list2=output END
LAF return_area_list INT_VAR from=2001 to=2999 RET to_check=list END
LAF iter_resource INT_VAR STR_VAR func=list_if_wav to_check ignore=are RET list3=output END
LAF iter_resource INT_VAR STR_VAR func=list_if_wav to_check=worldm25.wmp RET list4=output END

OUTER_SPRINT list "%list1% %list2% %list3% %list4%"

[eden]

In case, would this be compatible as to the EE edition of the games ?
(Or, it is "not needed" with it ?).

Seem they are quite some nice tools available by now, even if most release thread are quite empty (out of self support posts from the modders).

Full support that said, you are making what we enjoy, and so, support would be normal.

Thanks DavidW.