Detectable spells discussion

[khelban12]

Forgive me for hijacking the thread but i don't know where else to post it. SHS is down and this thread is the most recent thread about DS and has all the relevant people in it. Feel free to move it elsewhere if this is the wrong place for it.

Over the years i have noticed many DS versions being used by mods (some of them messing with stats when reordering them so that some mods cannot be installed because they can't find some stat they want).

I haven't followed the progression to the letter but it seems there are 3 versions used by major mods. At least from the git repos that i follow, i notice the following:

1) Many mods use a version named v3-20180512 (for example atweaks, rogue rebalancing)

2) Other mods use a version named v3.1 (for example item revisions). I am not a weidu expert but this version seems better to me than the 20180512. Some differences are that it uses some inline tables of spells and stats, the function names have the prefix ds_, a file that is used by the mod is named ag#dsal2.2da instead of ag#dsalt.2da, and it uses the function sort_array instead of SORT_ARRAY_INDICES.

3) spell revisions use version v3.95 with many changes by Ardanis which looks even better to me. Some differences are that the inline tables have many more values in them, some stats are named differently (for example 166 is changed from DMWW_SLOT_166 to MELEE_THACO_BONUS, etc).

Until here, everything is ok. I thought that eventually every mod will settle using the 3.95 version that spell revisions does but a few days ago, when i ran git pull on the ascension mod, i saw a commit that copied a new version of DS called v3.95_dw from the upcoming 32 version of SCS. One change is that instead of inline tables, it uses associative array. It doesn't look very different to me but i can't say for sure.

So i have some questions about this:

1) Is there a problem if a mod that is installed earlier uses an earlier (and presumably buggy) version of DS ? When a later mod is installed, does the DS run again (and presumably fix the bugs) or it sees that DS is installed and doesn't run (locally i have modified all the mods in game to use v3.95 but i ask out of curiosity).

2) Is there a central development location (let's call it "upstream") so that changes can be shared and avoid having in the future 2-3 codebases which have diverged very much from each other ?

Thank you for your time.

[Ardanis]

45 minutes ago, khelban12 said:

1) Many mods use a version named v3-20180512 (for example atweaks, rogue rebalancing)

I believe it wasn't updated to 3.1 or 3.95, hence it used its own fix for sorting problem discovered later:

37 minutes ago, khelban12 said:

sort_array instead of SORT_ARRAY_INDICES

...which is the same thing. I requested SORT_ARRAY_INDICES some years ago, and in the meantime coded a local function for DS.

40 minutes ago, khelban12 said:

3) spell revisions use version v3.95 with many changes by Ardanis which looks even better to me. Some differences are that the inline tables have many more values in them, some stats are named differently (for example 166 is changed from DMWW_SLOT_166 to MELEE_THACO_BONUS, etc).

Stat names have been updated in EE v2.0 to better reflect what they do, so MELEE_THACO_BONUS have become "official". DMWW_SLOT_xxx were mostly older "don't use it" names first introduced in SCS long, long ago.

53 minutes ago, khelban12 said:

I thought that eventually every mod will settle using the 3.95 version that spell revisions does but a few days ago, when i ran git pull on the ascension mod, i saw a commit that copied a new version of DS called v3.95_dw from the upcoming 32 version of SCS. One change is that instead of inline tables, it uses associative array. It doesn't look very different to me but i can't say for sure.

3.95 should have all the issues discovered since 2010 fixed and be sufficient for normal users who just want the spells flagged correctly.

The 3.95_dw shipped with SCS v32 has a bunch of updates and new functionalities added, which I have briefly checked by diagonal and they seemed fine. They're still new though, and at least one issue have been discovered and fixed between release candidates. It should be noted that David doesn't exactly qualify as "normal" user  so I can trust he knows what he's doing and why. One possible point of concern - his version may have some dependencies on SCS-specific environment, so I'd wait until updated SCS is out before trying to update other mods to it.

48 minutes ago, khelban12 said:

1) Is there a problem if a mod that is installed earlier uses an earlier (and presumably buggy) version of DS ? When a later mod is installed, does the DS run again (and presumably fix the bugs) or it sees that DS is installed and doesn't run (locally i have modified all the mods in game to use v3.95 but i ask out of curiosity).

It's fine. DS re-installs itself each time, and tries to fix known issues in previous versions if they can't be eliminated by mere re-install.

52 minutes ago, khelban12 said:

2) Is there a central development location (let's call it "upstream") so that changes can be shared and avoid having in the future 2-3 codebases which have diverged very much from each other ?

Hmm, probably not...

[khelban12]

Thank you for replying.

If my post can be read as implying that DavidW didn't know what he was doing then i apologize. At no time i even thought of something like that. I just wanted to avoid having two codebases being developed independently.

[Ardanis]

Well, DS has a long history of updates and maintainers... Ascension was the first one, and used direct file overwriting. Then Horred updated the stats it was using. Then either Cam or Bigg made it into re-installable patch. Then David further optimized it for SCS. Then SR started to shuffle effects between existing SPLs, so I took it over to make patching more dynamic and crossmod friendly. Then I added new EE features like spellstates. And now David's back in business and grabbed it back for SCS v32

[CamDawg]

3 hours ago, Ardanis said:

Then either Cam or Bigg made it into re-installable patch.

Yeah, this was part of the 'BG2 Fixpack should handle all of this junk' stage of development where we included DS and other stuff as part of the Super Happy Fun Lucky Modder Pack component. As I recall I just found what was considered to be the latest DS at the time and made it patch files instead of overwrite, not summon the Elder God*, pull data from an easily edited table, and be able to be installed multiple times without issue.

In the latest patch to Ascension I just used David's DS from SCS. The only SCS-dependent function it uses is David's custom IDS sorter instead of the one native to DS; I commented both out and used an unsorted stats.ids for Ascension. Neither sort function is currently satisfactory--David's sort function is superior, but its inclusion requires tracking down the 817 dependencies through 9,361 files** in SCS. The native DS ones nukes duplicate entries, which causes compile errors for scripts relying on the nuked entry (e.g. Ascension using SCRIPTINGSTATEx).

* This may be a slight exaggeration.
** This one may also be a slight exaggeration.

[DavidW]

Split this off from Lynx's thread. (Will comment substantively in a moment.)

[DavidW]

OK, I should probably be more responsible here. Doing a diff on my version and 3.95, I find that my version

1) adds a 'flexible_extended_stats' argument to the core function, defaulted to 1. If it's set to 1 then any stats entry with either opcode 318 or opcode 328 gets set to the appropriate value (318 on ToBEx, 328 on EE). 

2) rewrites the additions to core stats.ids so as to retain duplicates, without which some script compiling and similar was failing (Cam fixed a logic error here)

3) uses a bespoke SCS sort algorithm which, as Cam says, is better than the standard DS one (which I think I wrote too, long ago) but unfortunately introduces dependencies on SCS's function library

4) reroutes DS's marker files to using a bespoke file of mine. I'm not sure why I thought that was a good idea - I think it was part of a general move away from using in-game resources to generate marker files since that led to oddities elsewhere, but it's overkill here.

5) allows ids entries to be used in ds tables.  

6) fixes a bug in the resolve_ds_stat function (missing argument).

7) added a new function, ds_individual_spell, to add a ds state directly to a spell without having to go through a lookup table.

All this is backwards-compatible with the previous iteration, I think.

(4) is just silly and should be reverted. The rest I think is useful improvements to DS. I suggest I extract a version of the sort algorithm from SCS that's dependency-free, revert (4), retitle my version 3.96, and do a slightly better job of documenting the changes.

(@Khelban12: I didn't read you as saying anything rude.)

[Ardanis]

I'm pretty sure 3.95 should keep duplicates when sorting... During SoD development the QA department insisted the stats 176-182 shouldn't be used for detection because they also affect thief skills, so we skipped those in the EEs. But since mods have been long using those stats, I added a switch to keep them, as well as duplicate labels, so that both the existing mods reliant on those stats and any future ones expecting EE's names there wouldn't fail to compile.

 

PS I *think* it should keep them... not sure atm anymore.

Edited January 23, 2019 by Ardanis

[DavidW]

It doesn't keep them - specifically, the 'ds_sort_stats_ids' macro works by reading spell.ids into an associative array keyed by the number entries, and then outputting that array in order. So if there are duplicates, only the last is kept. Which idiot coded that?*

I came across it when installing SCS on top of Tweaks' modified-proficiency system started throwing errors. Tweaks leaves the old proficiency names in place pointing at the appropriate new proficiency, so that lookup in stats.ids still works, but DS was wiping those extra names.

 

* that would be me.

[Ardanis]

Nah, that'd be me, I re-coded v3 from scratch, as it was simpler than risking missing something in older code. Yours was only the idea which I copied, I think 

[Jarno Mikkola]

Ardanis, could you re-place the old v3, with the new version.