Spell selection issue? (sorcerers/character creation)

[Dan_P]

Anyone know anything about this? Like is there a file to edit if something gets messed up with the selection screen?

I noticed this with my own kits first, but also tested a few other kit mods, including Argent77's Chaos Sorcerer and the Tome & Blood sorcerer kits. All kits that normally have all spells available are affected.

When a bard, mage, or sorcerer kit is installed on a clean install or in front of most other kit mods, they have the full spell selection. However, if installed after certain other mods, something changes, and 1 or more schools get restricted. I'm still looking into it. Also, more than one opposition school type can be added. In a test install I ran for IWDEE, sorcerers installed near the end can't select evocation, illusion, or transmutation spells. The same kits installed early in the order can select all of those.

Note that kits installed early still have the full spell selection. They're unaffected. Only kits installed after whatever's causing the issue are affected. This is mainly an issue for sorcerers because mages and bards can still use all scrolls. The workaround I know of currently is to install the Tome & Blood "No opposition schools" component.

[jmerry]

This is extrapolation and speculation, but I think you're running into some very old and quirky code.

The kit IDs for the mage specialists are different from the kit IDs for other normal kits. Most kits (in the EE) start at 0x00004000 (trueclass) and count up; the last kit in standard BG2EE 2.6 is the priest of Tempus at 0x00004029. Mage kits, instead, have kit IDs with a single nonzero bit. Abjurers at 0x00000040, conjurers at 0x00000080, diviners at 0x00000100, enchanters at 0x00000200, and so on up to transmuters at 0x00002000. Specialists also use the same bits for their item unusability flags.

I suspect that arcane spell availability (to scribe, or to select as a sorcerer) is using that kit ID to check against the spell's exclusion flags. If the kit's ID includes that bit, the spell gets blocked. Which ... well, you're going to run into those offending bits pretty quickly if you count up. There's only room for 63 non-trueclass kits before you bump into the abjurer.

There are enough bits in the kit ID to work with to fit all of your kits. It's a 32-bit number, and only eight of those bits belong to the mage specialties. But I don't think the standard kit-adding routines are smart enough to skip over those eight bits and avoid the problem.

[Dan_P]

17 minutes ago, jmerry said:

This is extrapolation and speculation, but I think you're running into some very old and quirky code.

The kit IDs for the mage specialists are different from the kit IDs for other normal kits. Most kits (in the EE) start at 0x00004000 (trueclass) and count up; the last kit in standard BG2EE 2.6 is the priest of Tempus at 0x00004029. Mage kits, instead, have kit IDs with a single nonzero bit. Abjurers at 0x00000040, conjurers at 0x00000080, diviners at 0x00000100, enchanters at 0x00000200, and so on up to transmuters at 0x00002000. Specialists also use the same bits for their item unusability flags.

I suspect that arcane spell availability (to scribe, or to select as a sorcerer) is using that kit ID to check against the spell's exclusion flags. If the kit's ID includes that bit, the spell gets blocked. Which ... well, you're going to run into those offending bits pretty quickly if you count up. There's only room for 63 non-trueclass kits before you bump into the abjurer.

There are enough bits in the kit ID to work with to fit all of your kits. It's a 32-bit number, and only eight of those bits belong to the mage specialties. But I don't think the standard kit-adding routines are smart enough to skip over those eight bits and avoid the problem.

 

Thanks. I just checked. You got it right. I installed just enough kits to have a sorcerer before the abjurer bit, and another kit with the bit. Kit with the bit is restricted from illusion and transmutation in IWDEE (which are the abjurer restrictions). That also explains the evocation restriction I was getting. Conjurers in IWDEE are restricted from evocation.

[subtledoctor]

If that’s what it is, I made a function that can pad kitlist with extra blank kits to avoid the problematic kit id values!

The only other thought I have is could it be related to a UI tweak? Or to the OlvynSpells system that rewrites the spell selection process? I don’t know how those things work or if they could lead to this result, I’ve only read that they touch this screen. 

[Dan_P]

Just now, subtledoctor said:

If that’s what it is, I made a function that can pad kitlist with extra blank kits to avoid the problematic kit id values!

The only other thought I have is could it be related to a UI tweak? Or to the OlvynSpells system that rewrites the spell selection process? I don’t know how those things work or if they could lead to this result, I’ve only read that they touch this screen. 

Thanks, I'll try that out.

[subtledoctor]

If by try it out you mean insert that function into your local copies of various mods, you can find a more detailed write-up on how to do so here. Hopefully it is quite easy.

I was about to say “all of my kits already use it” but I just glanced at TnB and I have not added that to the sorcerer kits. I’ll put it on my to-do list. 

Edited November 18, 2022 by subtledoctor

[Dan_P]

1 hour ago, subtledoctor said:

If by try it out you mean insert that function into your local copies of various mods, you can find a more detailed write-up on how to do so here. Hopefully it is quite easy.

I was about to say “all of my kits already use it” but I just glanced at TnB and I have not added that to the sorcerer kits. I’ll put it on my to-do list. 

Hey, just an update to this. I tried the function but it didn't solve the issue for me. I also tested with your revised bards and multiclass kits components from Might & Guile, which I checked first and saw they included your function. They're also still affected by the spell restrictions.

 

I did make a few more observations with some kitids:

These kitids have abjurer restrictions:

0x00004049
0x0000405B
0x00004062
0x00004063
0x00004066

Your multiclass skald kit on this install has this:

0x0000406a

- it can't select, evocation, necromancy, illusion, or transmutation
The evocation and necromancy are from the kit, while the others are the abjurer restrictions in IWDEE.

 

On another install:

This kitids has only conjurer restrictions:

0x000040bf

These kitids have both abjurer and conjurer restrictions:

0x000040D1
0x000040D8
0x000040D9
0x000040DC

 

I think for now, instead of looking into it forever, I'm just going to add the recommendation for my own mods to remove school restrictions with the TnB component if there are issues.

Edited November 18, 2022 by Dan_P

[subtledoctor]

That’s… weird. I think you’re finding/suggesting that the exclusion applies if a mod kit’s value is bit-equal to the Abjurer or Conjurer (Or Diviner) values. But that would mean whole swaths of kit should be affected - not just 0x4040, but also 0x4041, 0x4042, through 0x404f… 0x4050 through 0x407f… right? Those are all bit-equal to 0x0040. 

If that’s the case, how has it never been noticed before?

Edited November 19, 2022 by subtledoctor

[subtledoctor]

2 hours ago, jmerry said:

There are enough bits in the kit ID to work with to fit all of your kits. It's a 32-bit number, and only eight of those bits belong to the mage specialties. But I don't think the standard kit-adding routines are smart enough to skip over those eight bits and avoid the problem.

Now I think about it, maybe ADD_KIT_EX should include this… assuming of course we confirm it actually works. Or maybe it’s outside the scope of the function? @argent77?

[jmerry]

On the flip side, this would seem to make it relatively easy to set up kits that deliberately have restricted schools for spell learning; specialist mages plus something else. Just ensure they have a kit ID with the right bits in it. Most kits would prefer to have all of those bits zero, because single-class kits might dual to mage even if they're not already arcanists.

Checking the code for add_kit_ex ... looks like it just picks the next number with no regard for the specialist bits, and no arguments that can influence the choice. The function writes that kit ID to the relevant 2DA and IDS files, and also returns it; anything that influences what that number should be is definitely in scope for the function. I'm thinking ... add extra optional arguments for the specialist bits. By default, they're off, but that can be changed so one or more are on.

[subtledoctor]

1 hour ago, jmerry said:

this would seem to make it relatively easy to set up kits that deliberately have restricted schools for spell learning; specialist mages plus something else

You can already do this by specifying this or that exclusion flag - no need to mess with the kit ID. 

1 hour ago, jmerry said:

Just ensure they have a kit ID with the right bits in it

IIRC kits have to get sequential kit IDs, in order. Messing with that messes with a bunch of other things. DavidW experimented with this very briefly when we discussed how to maximize the number of installable kits by back-loading multiclass kits. My recollection is that it did not work well.

1 hour ago, jmerry said:

Checking the code for add_kit_ex ... looks like it just picks the next number with no regard for the specialist bits, and no arguments that can influence the choice. The function writes that kit ID to the relevant 2DA and IDS files, and also returns it; anything that influences what that number should be is definitely in scope for the function.

It would have to incorporate something very like my function - the only way to deal with this AFAIK is to insert a dummy kit to occupy the problematic slot. Just not sure if it’s within the scope of the Add Kit function to basically run itself an extra time at those moments. 

[jmerry]

3 hours ago, subtledoctor said:

IIRC kits have to get sequential kit IDs, in order. Messing with that messes with a bunch of other things. DavidW experimented with this very briefly when we discussed how to maximize the number of installable kits by back-loading multiclass kits. My recollection is that it did not work well.

Considering the non-sequential inclusions in the existing KITLIST.2DA (mage specialists, barbarians, wild mages) and the non-sequential nature of KIT.IDS - the only two places where the kit IDs appear directly - I have to wonder just what went wrong. Maybe it's time to revisit and test things more?

I experimented a bit with existing kits in BG2EE - I didn't feel like actually adding kits or messing with kit IDs.

Setting the kit's "unusable" flags to the standard 0x00004000 that trueclass characters use did not allow specialist mages to use scrolls from their forbidden schools or to learn those spells at character creation. As a control test, that same change allowed a kensai to wear armor, helmet, and shield. If I also removed the unusability flag from a scroll, the specialist could then cast the scroll but not learn the spell.

Give a kensai the enchanter's unusability bit and dual him to mage? Now he gets an extra spell known, but no evocation spells are available. And he can't use a fireball scroll.

Give an enchanter the invoker's unusability bit instead of the usual enchanter bit? Character creation looks just like the enchanter; required to learn enchantment spells, can't learn evocation spells. Fireball scroll unusable, dire charm scroll usable both for learning and casting.

This is very weird.

Edited November 19, 2022 by jmerry

[Graion Dilach]

10 hours ago, subtledoctor said:

OlvynSpells system that rewrites the spell selection process? I don’t know how those things work or if they could lead to this result, I’ve only read that they touch this screen. 

ClassSpellTool doesn't affect spells which are not listed in it's system. On the other hand, it has tags for enabling/disabling so it might be possible that respecifying all the spells with it would fix this without dummy kits.

[argent77]

9 hours ago, subtledoctor said:

Now I think about it, maybe ADD_KIT_EX should include this… assuming of course we confirm it actually works. Or maybe it’s outside the scope of the function? @argent77?

If I understand it correctly, the issue of having specialist mage bits included in kit ids affects spell selection during character creation and spell selection for sorcerer kits at level up?

The former is probably too insignificant to warrant a rewrite of the ADD_KIT_EX function. The latter, however, is more serious.

The problem is that the engine supports only up to 256 single class kits, which would be in the kit id range [0x4000, 0x40FF]. The only specialist mage bits affecting this range are Abjurer (id 0x40, bit6) and Conjurer (id 0x80, bit7). That means kit ids from 0x4000 to 0x403F can be used by mage, bard and sorcerer kits to count as generalist mages by the engine, 0x4040 to 0x407F count as Abjurers, 0x4080 to 0x40BF count as Conjurers, and 0x40C0 to 0x40FF count as both Abjurers and Conjurers. Since the base game uses ids for vanilla kits up to 0x4029 there are only 22 more slots available for generalist mage kits.

I could probably improve ADD_KIT_EX to take care of this issue. Since the game engine expects kit ids in sequential order I would have to add placeholder kits to make it work. This will most likely introduce compatibility issues with the original WeiDU function ADD_KIT, although none of them are game breaking.

I don't know how many kit mods are using WeiDU's ADD_KIT, but it's probably still a great number mods. Is it really worth it to add such a radical change just to account for some (minor?) quirkiness of the game engine?

Edited November 19, 2022 by argent77

[jmerry]

I ran a more proper test in BG2EE. Pad kitlist out to 200, create new mage and sorcerer kits. Item usability bits as trueclass.

The sorcerer couldn't learn alteration (opposed to abjuration) or divination (opposed to conjuration) spells at character creation or level-up.

The mage couldn't learn alteration or divination spells at character creation, but she could from scrolls. She wasn't forced to learn and memorize spells of any particular school at character creation.

So, it's a relatively minor issue for mage kits (or kits that can dual to mage) and a major one for sorcerer kits. It's also definitely present across multiple IE games.

Edit: I should also note that this mostly default application of add_kit_ex resulted in a sorcerer kit that didn't have any proficiencies available. I think that the function looks for the base class's column in WEAPPROF.2DA if nothing is specified there ... but there's no SORCERER column so it zeros out everything. Sorcerer kits should use the MAGE column by default.

Edited November 19, 2022 by jmerry