Proposal for common table/something when adding spells to the game

[subtledoctor]

I have been thinking about how to get modders together and ensure compatibility among mods that add, change, or otherwise manipulate player-usable spells. This comes from reading this thread as well as several pleas for compatibility work, which have fallen on deaf ears. The beginning of the issue is this:

• The number of spells available to sorcerers/shamans at level-up, and to mages/bards/clerics/druids at character generation, is limited to 50 per level.
• These spells share a naming scheme (SPWI101 through SPWI150, SPWI201 through SPWI250, etc., and ditto with the SPPR prefix), and they all get recorded in SPELL.IDS and get IDS numbers for reference (1101 through 1150, 1201 through 1250, etc. and 2101 through 2150, etc.).
• These spells are all subject to handy reference and manipulation via Weidu's RES_NUM_OF_SPELL_NAME, and similar functions. A typical use of that function involves first checking whether the spell name exists in SPELL.IDS, and then looking up its ResRef (filename).
• The vanilla game uses up roughly 30% to 50% of the available slots at any given spell level; mods add more.
• Mods that add more generally do so via Weidu's ADD_SPELL function, since it handily makes the spells available for lookup and use in game with all the above characteristics, automatically finding the next available slot and assigning a compatible filename. However, if there are no available slots, the ADD_SPELL function will fail, and it will actually cause the entire mod install to fail.  (There are reasons for that; let's not litigate it here.)

So, now there are several mods that add spells this way, and if a player uses enough of them, the later ones are subject to install failures. Nobody likes to see that.

Recently, Olvynchuru has added a new mechanism in his mod that applies a UI modification to the sorcerer/character generation spell-picking screens. My basic understanding is, these modified screens allow you to pick from any of the usual spells, and additionally from spells added by his mod. This solves the problem of his spells being excluded; but it introduces several new issues:

• I don't know whether his system has hooks to allow other modders to have their spells included in his modified spell-picking screens; this could especially be an issue for mods installed after OlvynSpells.
• Mods that try to comb through all player-available spells will generally use one of a couple methods: comb through SPELL.IDS; use regexp to match the "SPWI###" and "SPPR###" filenames; (for arcane spells) checking all scrolls from which a spell can be learned, and compiling the spells that can be so learned; etc. OlvynChuru's mechanism will render several of these methods non-functional. I'm not sure how you could accomplish this with spells added by OlvynChuru's system, especially if other mods start using their own variants of the system.
• There is no equivalent of a RES_NUM_OF_SPELL_NAME function available for spells added by his system.

So, me personally, I am not about to add a whole new section of code to my mod just to handle Olvyn's spells in addition to the ones added by traditional means. Set aside that it would be non-trivial and bug-prone; even if I put in the effort, my mods would still not be compatible with the next mod that comes along and adds spells in this new way. (I may understand that @Artemius I may be adapting his mod(s) to such a system...?) Maintaining compatibility will be untenable if more mods do this sort of thing.

I think the best solution is, more or less, to make a new and expanded version of SPELL.IDS. Call it MODSPL.IDS. Maybe begin with the full contents of SPELL.IDS (or perhaps just the arcane and divine spells, since this issue doesn't really affect innates). Then any mod adding new spells that may go past SPWI150 can 1) check whether MODSPL.IDS exists; 2) if it doesn't exist, create it; and 3) append information for the new spells to it. These steps can be tucked into a nicely portable function, usable by any mod. Then, on the other side, mods that need to know what spells are in the player's game can 1) check whether MODSPL.IDS exists; 2) if it does exists, do the business with respect to spells in that file; 3) if it does not exist, do the business with respect to spells in SPELL.IDS.

MODSPL.IDS can, for mod-added spells, simply list the ResRef in place of the traditional IDS number. This way such spells can uses modder prefixes, but those spells and their modder prefixes would be easily available from a central list to other mods and UI functions, etc.  We might also want variants of the RES_NUM_OF_SPELL_NAME functions, but those should be pretty simple to make and will likewise be nicely portable.

In conclusion: I honestly don't even kn ow if this is necessary. Maybe it's more trouble than it's worth. But I'm curious to hear the thoughts of other people.

Context: I'm tinkering with a total-conversion-ish 5E spellcasting mod that will want to know all spells available to each class. The current beta version of the 5E system (my Arcanist kit) does not work with Olvyn's Spells. If there is consensus for something like what I set out in this post, I would love to bake it into my full mod.

Edited February 24, 2021 by subtledoctor

[Jarno Mikkola]

9 minutes ago, subtledoctor said:

Recently, Olvynchuru has added a new mechanism...

And there's no way to adopt this into a better defined / more mod-maker friendly system that just takes the old ADD_SPELL -like weidu fuctionality and expands the cheese out of it, so that everyone should use it ?

It doesn't need to be OlvynChuru that directs this, and I am actually hoping someone that is good at making tutorials to do such. 

As I undestand it, the ADD_SPELL is an "easy" weidu function, relatively... maybe we(well, spell mod makers) just need to learn a medium one too.

Yes, I know this is all EE game exclusive, and require the EEex and blah, but seriously these games have been there for a quite long time now and one ...

17 minutes ago, subtledoctor said:

Call it MODSPL.IDS

This is a horrible idea... if it's not made by Beamdog, but one modder alone.

Meaning that we should just make every spell adding mod adopt a system that allows everyone of them equaly well access to the targets.

[Endarire]

I am whole heartedly in agreement that a common system for adding the hundreds or thousands of modded spells from Infinity Engine games should be made into an EE- and EET-compatible mod!

As for how best to achieve this, I'll let others speak first.

(PS: Thankee for reading that thread I wrote and you linked!  Alleluia!)

[Jarno Mikkola]

46 minutes ago, subtledoctor said:

These spells are all subject to handy reference and manipulation via Weidu's RES_NUM_OF_SPELL_NAME, and similar functions. A typical use of that function involves first checking whether the spell name exists in SPELL.IDS, and then looking up its ResRef (filename).

...

There is no equivalent of a RES_NUM_OF_SPELL_NAME function available for spells added by his system.

This is because the ADD_SPELL commands Resource_binding was married into it and weidu afterwards, probably by multiple requests. I bet that Wisp has not had the time to familiarize himself into say ouh the OrvynSpells 4760 lines long .tp2 code, to make a good suggestion on what to improve or overhaul the code with etc. See the last 200 lines of code for example being completely quoted out. It not really his job either.

Edited February 24, 2021 by Jarno Mikkola

[subtledoctor]

1 minute ago, Jarno Mikkola said:

This is because it was married into weidu afterwards, probably by multiple requests. I bet that Wisp has not had the time to familiarize himself into say ouh the OrvynSpells 4760 lines long .tp2 code, to make a good suggestion on what to improve or overhaul the code with etc. See the last 200 lines of code for example being completely quoted out.

I don't think any of Weidu's existing functions need to be altered, or that Wisp needs to be bothered. I envision something like a portable pack of functions that any modder can use, so that 1) they don't have to write the code from scratch, and 2) inter-mod compatibility will be baked in, with no extra effort.

[Jarno Mikkola]

1 minute ago, subtledoctor said:

I don't think any of Weidu's existing functions need to be altered, or that Wisp needs to be bothered. I envision something like a portable pack of functions that any modder can use, so that 1) they don't have to write the code from scratch, and 2) inter-mod compatibility will be baked in, with no extra effort.

I agree completely that this would be handy. But it can be added into weidu too. And no, it would not be alteration of the ADD_SPELL, but some other say ADD_SPELL_SPECIAL function, or just the ASS -function.

[kjeron]

Use ".2DA", an IDS extension may cause confusion, especially if someone tries to run IDS_OF_SYMBOL on it.

Even if you don't hook into Olyven's system with his function, with a common ResRef->LABEL list, it would be easy to run your own parallel to it without either interfering.

You can rewrite RES_NUM_OF_SPELL_NAME (macro/function/action/patch) in your ALWAYS block to include an alternate spell table without breaking it's current usage.

[subtledoctor]

1 hour ago, kjeron said:

Use ".2DA", an IDS extension may cause confusion, especially if someone tries to run IDS_OF_SYMBOL on it.

Very good point. 2da is just as good for these purposes. 

1 hour ago, kjeron said:

You can rewrite RES_NUM_OF_SPELL_NAME (macro/function/action/patch) in your ALWAYS block to include an alternate spell table without breaking it's current usage.

If necessary, I was just thinking about copying and pasting it into a new “RES_NUM_OF_MODSPL_NAME” function, only changing the targeted table. Put it in a function library along with other stuff here, and then any mod can include and use it. 

(Though now that I think of it, that’s probably unnecessary. If we just make a big table with IDS names and corresponding ResRefs, then READ_2DA_ENTRY should be sufficient in ~99% of cases.)

[Galactygon]

I have started a topic over 10 years ago in an attempt to set a commonly-agreed list of IDS_OF_SYMBOL names, originally meant for mega-mods. Heck, this was before the days of the EEs. Time really flies.

http://www.shsforums.net/topic/46717-add-spell-in-spell-mods/

Not much has come out of that since the traditional mega-mods have long been abandoned. It's certainly a good idea to have agree on a list of names so that mod-added ADD_SPELLs replace each other in order of installation without generating doublettes.

[subtledoctor]

1 hour ago, Galactygon said:

I have started a topic over 10 years ago in an attempt to set a commonly-agreed list of IDS_OF_SYMBOL names, originally meant for mega-mods. Heck, this was before the days of the EEs. Time really flies.

I know! That thread is largely my inspiration here. And I think that was actually very useful - even all this time later, there was some discussion of it recently when Spell Revisions changed over to using ADD_SPELL, as it contains some of the same spells as IWDification etc. (Btw that's a good thread for anyone who is adding spells to peruse - @Angel you might want to take a look, if you were not already aware of it.)

Anyway, just writing out the posts here have clarified my thoughts a bit. Like, logistically, twinning this with spell.ids could be problematic. What if a spell mod like Olvyn's copies spell.ids to the new "modspls.2da" or whatever; and then a different mod comes after and adds new stuff to spell.ids? Now modspls.2da will be out of date.

Further: what would the format of the 2da table be? Are spell pack mods expected to create pseudo-IDS names for their spells, when the spell will not go into spell.ids and the pseudo-IDS name will not and cannot  be used  with stuff like IDS_OF_SYMBOL? What would the point of that even be? 

Although conversely, if the spell pack were to use ADD_SPELL then it would have to add an IDS name anyway, so of the point is to add spells usable by sorcerers/shamans, then it is no extra effort.

So I guess this might look something like this:

ACTION_TRY
  ADD_SPELL ~mymod/spells/d5nuspl.spl~ 2 5 ~WIZARD_EVARDS_BLACK_TENTACLES~
WITH
  DEFAULT
    ADD_MOD_SPELL ~mymod/spells/d5nuspl.spl~ 2 5 ~WIZARD_EVARDS_BLACK_TENTACLES~
END **

...and our new "ADD_MOD_SPELL" function would write the IDS name and ResRef to modspls.2da, in addition to doing whatever hocus-pocus is necessary for Olvyn's UI modification to capture it.

Then a mod that comes along later to poll wizard spells can do something like:

COPY_EXISTING ~spell.ids~ ~override~
  COUNT_2DA_COLS cols
  READ_2DA_ENTRIES_NOW ~r2en_spls~ cols
  FOR (row = 0; row < r2en_spls; ++row) BEGIN
    READ_2DA_ENTRY_FORMER ~r2en_spls~ row 0 ids_num
    READ_2DA_ENTRY_FORMER ~r2en_spls~ row 1 ids_name
    PATCH_IF (~ids_name~ STRING_CONTAINS_REGEXP ~WIZARD_~ = 0) BEGIN
      LPF RES_NUM_OF_SPELL_NAME STR_VAR spell_name = ~%ids_name%~ RET spell_res END
      PATCH_IF (VARIABLE_IS_SET ~%spell_res%~) BEGIN
        PATCH_IF (FILE_EXISTS_IN_GAME ~%spell_res%.spl~) BEGIN
          SPRINT $wiz_spells(~%spell_res%~)~1~
        END
      END
    END
  END
BUT_ONLY

ACTION_IF (FILE_EXISTS_IN_GAME ~modspls.2da~) BEGIN
  COPY_EXISTING ~modspls.2da~ ~override~
    COUNT_2DA_COLS cols
    READ_2DA_ENTRIES_NOW ~r2en_spls~ cols
    FOR (row = 0; row < r2en_spls; ++row) BEGIN
      READ_2DA_ENTRY_FORMER ~r2en_spls~ row 0 mod_res
      READ_2DA_ENTRY_FORMER ~r2en_spls~ row 1 ids_name
      PATCH_IF (~ids_name~ STRING_CONTAINS_REGEXP ~WIZARD_~ = 0) BEGIN
        PATCH_IF (FILE_EXISTS_IN_GAME ~%spell_res%.spl~) BEGIN
          SPRINT $wiz_spells(~%spell_res%~)~1~
        END
      END
    END
  BUT_ONLY
END

ACTION_PHP_EACH wiz_spells AS res => ind BEGIN
  [do stuff]
END

So I guess the main things of value would be 1) write up that ADD_MOD_SPELL function and associated functions, nice and portable so any spell pack mod can include them and use them without much extra effort; and 2) having a consensus on the name and form of my hypothetical "modspls.2da" file, so that any other mod will have a stable and consistent place to look for such spells.

** (Or whatever the appropriate syntax is for TRY... I have never fully wrapped my head around it. I wish Weidu had something a bit simpler like TRY [this] ON_ERROR [that] END...)

Edited February 25, 2021 by subtledoctor

[kjeron]

2 hours ago, subtledoctor said:

So I guess the main things of value would be 1) write up that ADD_MOD_SPELL function and associated functions, nice and portable so any spell pack mod can include them and use them without much extra effort; and 2) having a consensus on the name and form of my hypothetical "modspls.2da" file, so that any other mod will have a stable and consistent place to look for such spells.

2) My vote is for "ADDSPELL.2DA".

I think it would be better to store them ids => res instead of res => ids, so you can do the following COPY_EXISTING:

ACTION_PHP_EACH wiz_spells AS ids => res BEGIN
      // ACTION
END

COPY_EXISTING ~%wiz_spells_WIZARD_ARMOR%.SPL~ override
      // PATCH
BUT_ONLY IF_EXSITS

Scanning filenames, then finding it's IDS, would be better than scanning IDS, and finding it's resref.

SPWI101 will be offered to a wizard or sorcerer (barring specialist exclusions), no matter what label 2101 has in SPELL.IDS.

COPY_EXISTING_REGEXP ~^SPWI[1-9]\([0-4][0-9]\|50\)\.spl$~ override
	PATCH_IF	NOT FILE_CONTAINS_EVALUATED (~HIDESPL.2DA~ ~^%SOURCE_RES%[ %TAB%]+1~) BEGIN
		LPF    NAME_NUM_OF_SPELL_RES    STR_VAR    spell_res = $SOURCE(RES)    RET    spell_name    END
		SPRINT    $wiz_spells($spell(name)) $SOURCE(RES)
	END
BUT_ONLY

COPY_EXISTING ~ADDSPELL.2DA~ override	READ_2DA_ENTRIES_NOW READ 2
	FOR (i = 1; i < READ; ++i)    BEGIN    SPRINT $wiz_spells($READ(~%i%~ 1)) $READ(~%i%~ 0)    END
BUT_ONLY    IF_EXISTS

 

[Angel]

19 hours ago, subtledoctor said:

Btw that's a good thread for anyone who is adding spells to peruse - @Angel you might want to take a look, if you were not already aware of it.

Although it is certainly interesting and I am keeping a curious eye on the developments, my interest in it is only marginal for several reasons: 1) Even with several spell-providing mods installed, I don't come anywhere near the 50 spells per level limit (barely halfway there), 2) I do not use Eex, and cannot use it even if I wanted to unless Linux support was added while I wasn't looking, and 3) Not using spell.ids means huge changes in AI scripting that will not be done overnight.

It would likely also break backwards compatibility with BGT, but to be honest I have already been sorely tempted to drop non-EE support at this point as it is almost as much a pain as adding vanilla BG1 support, so that wouldn't be a huge loss to me. ^^

Anyway, even if a new function became available, it would have to be integrated in SFO for my mods to use it, or I would have to do massive rewriting, neither of which I look forward to.

[Jarno Mikkola]

20 hours ago, kjeron said:

SPWI101 will be offered to a wizard or sorcerer (barring specialist exclusions), no matter what label 2101 has in SPELL.IDS.

Yeah, but that's because the spin101.spl is not flagged as non-wizard/sorcerer one in the .spl files offset 0x001e. Yes, technically that's specialization exclusion, but it's a whole class wide one in bit 14 and 30&31. And they can be undone by kits own GA_spellname -powers in it's clabxxxx.2da file. Or with a scroll item, that uses the opcode 147.

[subtledoctor]

6 hours ago, Angel said:

Although it is certainly interesting and I am keeping a curious eye on the developments, my interest in it is only marginal for several reasons: ...

Sorry, I meant it’s worth taking a look at Galactygon’s linked thread. Just for nitty-gritty issues like, is it “WIZARD_EVARD_BLACK_TENTACLES,” or “WIZARD_EVARDS_BLACK_TENTACLES?” If two mods use slightly different names, it can be problematic, in that players would get duplicative spells. That thread shows what consensus has been achieved so far; and if you are adding more spells that might conceivably be added by other mods (core rulebook stuff) you can post it there so others can match what you did.

[subtledoctor]

Incidentally, I had not seen this construction before:

Quote

LPF NAME_NUM_OF_SPELL_RES STR_VAR spell_res = $SOURCE(RES) RET spell_name END

I would have used "EVAL ~%SOURCE_RES%~" there... is "$SOURCE(RES)" different?