Errors while installing reduced reputation increase

[ScuD]

Never had false positives for DLG in NI before (I'm not saying it is bugless).

But 2 .D files (before and after this component) after decompilation differ: before and after.

[the bigg]

file corruption

Apparently WeiDU dumps some useless stuff in the compiled DLG if there's an empty DO action, as vanilla or post-fixpack scsarles.dlg does (bottom response of state 15). Since this behavior was already present in WesDU times, it likely is harmless; moreover, I'm not familiar with the relevant section of code, so I won't fix it unless you can prove it creates problems.

 

I can say significantly less files are being patched (with 22808). Something is not right.

That's because DW's old code (de)compiled all scripts rather than just the ones that actually use ReputationInc(), and DECOMPILE + COMPILE is not guaranteed to be idempotent.

[ScuD]

...so I won't fix it unless you can prove it creates problems

As long as your code works faster and better there's no need for the fix. And also for recompiling all the files in the game.

I'll test it some more and report to you in the PPG forums if I see more problems.

[DavidW]

...so I won't fix it unless you can prove it creates problems

As long as your code works faster and better there's no need for the fix. And also for recompiling all the files in the game.

 

I remain unclear why decompiling and recompiling all the files in the game is a bad thing. Is it about install time? (It takes 60 seconds on my install.) Is it about worrying about creating bugs? (But if some script decompiles and recompiles to something functionally (and not just harmlessly) different from what it started out as, something is wrong with the script or with WEIDU, and that will cause problems anyway, independently of the decompile-recompile in SCS.)

[Miloch]

I remain unclear why decompiling and recompiling all the files in the game is a bad thing.

Why would it be a good thing? Why would someone do it if it could be avoided?

Is it about install time? (It takes 60 seconds on my install.)

A minute for one component isn't exactly trivial, and I'll bet anything you could multiply that at least by 10 on a BWP-type install, possibly 100 or more on a slower machine. The last time I did a full BWP install on an external drive, SCS1 and 2 were amongst the longest installs (several hours each perhaps). Truth be told, Aurora was a big offender too. If I find ways to reduce the install time, I try to use them. The code revision above seems an easy call, since you only need to decompile files you're actually changing.

[DavidW]

I remain unclear why decompiling and recompiling all the files in the game is a bad thing.

Why would it be a good thing? Why would someone do it if it could be avoided?

On one interpretation of "if it could be avoided", that's true for absolutely any change. (Why modify the game AI if it can be avoided?)

 

The impression given is that there's something particularly problematic about this move as compared to others, which is why I was curious as to what the problem was. It sounds like the problem is just that it takes too long, which is fine, but it would have been helpful to have made that clear up front.

 

Is it about install time? (It takes 60 seconds on my install.)

A minute for one component isn't exactly trivial, and I'll bet anything you could multiply that at least by 10 on a BWP-type install,

There are seriously ten times as many scripts added by BWP mods as there are in the entire original game? I find that hard to believe.

 

possibly 100 or more on a slower machine.

If your machine runs ten times slower than mine, it probably won't handle SCS anyway, so I'm not much moved.

 

The last time I did a full BWP install on an external drive, SCS1 and 2 were amongst the longest installs (several hours each perhaps). Truth be told, Aurora was a big offender too. If I find ways to reduce the install time, I try to use them. The code revision above seems an easy call, since you only need to decompile files you're actually changing.

 

Sure, I don't object to shortening run time when it's straightforward to do. (And this case is an example where it is straightforward to do.) But (i) SCS run time (at least on a normal (non-BWP) install) is totally dominated by other processes anyway; (ii) I don't generally try hard to reduce install time at the cost of logical clarity in the code, because that kind of thing generates bugs, and God knows I find it hard enough to keep bugs under control as it is. SCSII, in toto, takes about twenty minutes to install on my machine, and that figure wasn't radically different last time I tried it on an oldish laptop.

[the bigg]

It bears stating that people who install on external drives get what they deserve.

 

Moreover, 90% of scsii's install time is spent compiling SSL files and the resulting gigantic BAF files, so optimizing the remaining 10% of the install time doesn't make any sense from an engineering POV.

 

That said, a full BWP installs a scary amount of files, and even 10x isn't out of the question: a vanilla chitin.key contains 41793 files, vs. a BiG Expert install I made some months ago which had 376541 files.

[Miloch]

On one interpretation of "if it could be avoided", that's true for absolutely any change. (Why modify the game AI if it can be avoided?)

Your mod sets out specifically to change the AI, so it can't rightly avoid that. Let me rephrase the second question then: "Why would you decompile and recompile *all* scripts in the game if you're only modifying a small subset of them?"

There are seriously ten times as many scripts added by BWP mods as there are in the entire original game? I find that hard to believe.

There are at least 3 or 4 times as many, according to my (old) install. But that's just the number - I have no doubt people have also bloated existing scripts to be far larger than the original ones.

If your machine runs ten times slower than mine, it probably won't handle SCS anyway, so I'm not much moved.

I wasn't looking to run it, just install it, along with every other mod. That's how I do compatibility testing before releasing a mod, and I'm not about to eat up my limited internal drive space with several gigabytes of BWP.

Sure, I don't object to shortening run time when it's straightforward to do. (And this case is an example where it is straightforward to do.) ... I don't generally try hard to reduce install time at the cost of logical clarity in the code

I'm not sure I follow this, since you're arguing in favour of the specific case but in apparent disfavour of such cases in general. I don't think adding a few lines of code is going to impair "logical clarity" beyond recognition. Personally, I'd take a hit to code clarity (which benefits mainly one person - the coder) over install time (which benefits not just the coder but everyone else installing the mod). What's odd is to see the bigg argue against this also, when he's the one who wrote an improvement (though maybe his code doesn't work).

SCSII, in toto, takes about twenty minutes to install on my machine

That's a long time to install one mod if you ask me. WeiDU actually handles many complex tasks quite quickly. What happens if you want to tweak and/or test something that requires a reinstall (possibly several reinstalls until you get it right)?

[DavidW]

Sure, I don't object to shortening run time when it's straightforward to do. (And this case is an example where it is straightforward to do.) ... I don't generally try hard to reduce install time at the cost of logical clarity in the code

I'm not sure I follow this, since you're arguing in favour of the specific case but in apparent disfavour of such cases in general.

I'm arguing against doing it when it's complicated or messes with the clarity of the code. I'm conceding that's not true in this case (though equally, I'm unrepentant about having coded it it that way; it's just that there's a better way.)

Personally, I'd take a hit to code clarity (which benefits mainly one person - the coder) over install time (which benefits not just the coder but everyone else installing the mod).

I'm unmoved by that, because

(i) it's not true that logical clarity doesn't benefit others. I have a finite, fixed time to code, so anything that makes things take longer for me is going to either lead to more bugs or to less content.

(ii) if it were true that it only benefits me, I don't care. I'm totally relaxed about doing this as a hobby I enjoy; I don't do it as a social duty.

 

 

SCSII, in toto, takes about twenty minutes to install on my machine

That's a long time to install one mod if you ask me. WeiDU actually handles many complex tasks quite quickly. What happens if you want to tweak and/or test something that requires a reinstall (possibly several reinstalls until you get it right)?

Then it takes lots of time. But since, as TheBigg pointed out, 90% of that time is SSL and BAF compiling, so you're stuck with it in any case. (I massively prioritise in-game speed over install-time speed.)

[ScuD]

JFYI, in my megainstall the total number of files (BCS+DLG) is 14,241. After the installation the number of changed files is only 157 (2 BCS and 155 DLG).

So I personally consider thebigg's code better than doing a recompilation of the whole bunch of dialogs/scripts.

[the bigg]

In this case making it faster is free (well, if you don't consider the 30 minutes needed to learn of the new WeiDU commands and add them to your code), so there's no reason not to; I take issue when you write an ACTION_BASH_FOR with 300+ items because you adamantly insist that "regexp is slow".

[DavidW]

In this case making it faster is free (well, if you don't consider the 30 minutes needed to learn of the new WeiDU commands and add them to your code), so there's no reason not to; I take issue when you write an ACTION_BASH_FOR with 300+ items because you adamantly insist that "regexp is slow".

 

Yes, that's exactly the kind of distinction I had in mind. I am (to repeat ad nauseam) not disputing that TheBigg's code is smoother than my original version (actually the version in SCSII v17 isn't quite either, because I didn't have time to learn some of the new v228 commands, but it's TheBigg's in spirit). But I do find from time to time that people have odd and unjustified shibboleths about certain forms of coding (the last example was the discussion at SHS about cluttering the override) and so it's interesting to get clearer on why. I think Miloch's comment that code clarity is almost-always to be sacrificed to install speed points at quite a big philosophical difference.

[Miloch]

I'm arguing against doing it when it's complicated or messes with the clarity of the code. I'm conceding that's not true in this case (though equally, I'm unrepentant about having coded it it that way; it's just that there's a better way.)

I don't think anyone was flaming you for having coded it the way it was, when there wasn't a significantly better way at the time, so no need to get defensive. However, it seems like most of your objections came after being presented with an improved alternative. You've conceded it makes more sense; fine; why not let us leave it at that?

(I massively prioritise in-game speed over install-time speed.)

(SCS has components that improve in-game speed? 'Faster Bears' maybe?)

In this case making it faster is free (well, if you don't consider the 30 minutes needed to learn of the new WeiDU commands and add them to your code), so there's no reason not to; I take issue when you write an ACTION_BASH_FOR with 300+ items because you adamantly insist that "regexp is slow".

This, on the other hand, looks like a flame (directed at me no doubt). ACTION_BASH_FOR still uses regexp if I'm not mistaken. But WeiDU can handle passing 300+ items to an ACTION_FOR_EACH, and handle it fairly quickly, so what's wrong with it? (At least, it can now; it used to barf around ~255 items but that only meant you needed a whole *two* AFEs to handle ~300 items.) I do know that regexp *was* slow for complex tasks - maybe still is. IIRC, when I first mentioned it, you agreed and blamed it on OCaml. Maybe you've managed to make some improvements; maybe code such as the above would help. If so, that's great, and I might even look to using it in my own code (if I can remember after having to address two dozen other things). No doubt it could improve something like Aurora's Quest Gold component, but when we wrote it, both GeN1e and I got fed up with the time it took to test it using regexp. Yes, it decompiled and recompiled every script and dialogue in the game also, but under quite a complex set of conditions. So decompiling and recompiling all DLGs on your BG2 with 1 change doesn't compare at all. If you don't believe it was slow, change the A_F_E in ag_quest_gold.tpa to regexp and slap it on a realistic install scenario. By that, I mean a number of large mods before it (a la BWP) because let's face it, that's what a good number of folks are using these days, and for the most part, they can, thanks to compatibility efforts.

 

And if you have issue with that, that's what I would call "unjustified shibboleths about certain forms of coding." This wasn't some "philosophical" scenario, but one in which we made a definite improvement due to the previous method being rather intolerable (to us as coders/testers, even if we didn't care about anyone else).

[the bigg]

I don't think anyone was flaming you for having coded it the way it was, when there wasn't a significantly better way at the time, so no need to get defensive.

Technically, you could write something that looked 95% like my code using WesDU (see: COUNT_REGEXP_INSTANCES, or even IF) - the only thing you couldn't have with earlier versions of WeiDU was the ACTION_TRY bit (and the guarantee you're not leaving DECOMPILEd but not reCOMPILEd in the override).

 

I do know that regexp *was* slow for complex tasks - maybe still is.

I think the big delay you had BEFORE printing 'Copying and patching XYZ files' was fixed in some early version; moreover, if you use PATCH_IF filtering there shouldn't be noticeable difference between C_E_R and A_F_E I_F_E C_E.

[DavidW]

I'm arguing against doing it when it's complicated or messes with the clarity of the code. I'm conceding that's not true in this case (though equally, I'm unrepentant about having coded it it that way; it's just that there's a better way.)

I don't think anyone was flaming you for having coded it the way it was, when there wasn't a significantly better way at the time, so no need to get defensive.

 

I wasn't intentionally being defensive. Actually, though, the relevant speed-up bit of the code (checking for whether the file contains the relevant data before decompiling it) can be done with pre-228 WEIDU (and is done that way in SCSIIv17). I probably couldn't have worked out the uncompiled term corresponding to ReputationInc without help, though.

 

However, it seems like most of your objections came after being presented with an improved alternative. You've conceded it makes more sense; fine; why not let us leave it at that?

Because I'm interested in general principles of mod design, and because I have a philosopher's habit of being more interested in the strengths and weaknesses of the arguments for a conclusion than in the conclusion itself.

 

(I massively prioritise in-game speed over install-time speed.)

(SCS has components that improve in-game speed? 'Faster Bears' maybe?)

SCS has components that improve in-game speed relative to what they could do to it. Case in point: the "Smarter mages" component of SCSII generates about 90 mage scripts, averaging 10,000+ lines each. Compiling those scripts SSL->BAF and BAF->BCS takes quite a while (about 10 minutes on my relatively fast computer). I could get that 10 minutes down to 20 seconds by only having one mage script for all mages (that's how Tactics does it, and it's how earlier versions of SCSII did it.) But the benefit of spending the 10 minutes is that in-game mage scripts are much shorter (usually 30%-60% as long) as they would be on the old method, and that leads to a noticeable decrease in in-game lag.

 

But WeiDU can handle passing 300+ items to an ACTION_FOR_EACH, and handle it fairly quickly, so what's wrong with it? (At least, it can now; it used to barf around ~255 items but that only meant you needed a whole *two* AFEs to handle ~300 items.) I do know that regexp *was* slow for complex tasks - maybe still is.

 

This is quite a good example of my "logical clarity" issue. It's possible a given component actually wants to patch a particular set of 117 files, for reasons specific to each file. Even if it so happens that there's some easy REGEXP that picks them out, it would misrepresent the structure of the code to use COPY_EXISTING_REGEXP. Conversely, if the idea is that all files with feature X should be patched, it's logically much clearer to use REGEXP to pick out those 117 files, rather than list them. Of course there are tradeoffs (replace 117 with 2, and things change) but I take it that's the baseline. (And, to reiterate: intelligible, clear code isn't just an indulgence: it correlates strongly with speed of writing and with avoiding bugs.)

 

a realistic install scenario. By that, I mean a number of large mods before it (a la BWP) because let's face it, that's what a good number of folks are using these days, and for the most part, they can, thanks to compatibility efforts.

 

That's an interesting comment, actually. Personally I don't see a BWP megamod install as my primary-intended SCS(II) audience. I'm very happy if megamodders want to and can use SCS, and I'm happy to make minor adjustments to accommodate them, but when I code SCS I generally have in mind a significantly smaller mod base (in particular, I don't usually have in mind individual megamods like TDD or CtB). I think it's a very unfortunate corollary of BWP that it's led to people preferring quantity over quality, given the sharp variations in the latter in this or any modding scene.

 

Perhaps it's worth saying again at the end of this that I'm pursuing it because I think it's an interesting discussion, not because I'm feeling annoyed or defensive.