Rare selection sounds

[Miloch]

The frame rate component seems to be choking when another mod has already copied a sound clip to the override. I suspect this is because they're decompressed WAVs rather than WAVCs (for example kivan35, 36, 37 from the Kivan mod here at G3). Apparently, the engine recognises both formats. I guess I'll have to put in a check for that somehow. I suppose it'd consist of reading the header and skipping the file (or removing it from the array, if that's possible) if it's not the right format. Any ideas, let me know.

[Bearwere]

The frame rate component seems to be choking when another mod has already copied a sound clip to the override. I suspect this is because they're decompressed WAVs rather than WAVCs (for example kivan35, 36, 37 from the Kivan mod here at G3). Apparently, the engine recognises both formats. I guess I'll have to put in a check for that somehow. I suppose it'd consist of reading the header and skipping the file (or removing it from the array, if that's possible) if it's not the right format. Any ideas, let me know.

yep, reading the header will do (though I don't understand why Kivan is shipped with decompressed .wavs in the first place). I'm not sure about arrays, I haven't got to mess much with them.

[cmorgan]

I am a little ways behind you guys - but if you are talking about things not being in WAVC format in he override, there are very few mods which use WAVC. Almost all use oggdec to compress to .ogg, and then best practice these days is to decompress to .wav in the mod folders and move uncompressed .wav to override.

 

There are even reports of area ambients needing decompression to standard .wav files (I can't remember the full discussion).

 

If I understand what you are saying enough, it seems like you could simply batch compress any preexisting raw .wav in override - the bigg has code (and the processing .exe needed) in Amber (certainly in the linux distribution) where he does this for all of her sounds.

 

(side note - players report installs with a ton of NPCs with soundfiles cause lag. I wonder if the engine would be happier with WAVC format instead? I can't imagine everyone has been putting raw .wav files in place when the WAVC format has been available and would reduce the size of the override, unless someone had determined that .wav was better in some way - may have to go thread-archeology-searching, or beg Smoketest and devSin to weigh in...)

[the bigg]

I also don't understand why the IESDP doesn't detail the ACM format, because it's a "proprietary" format. Heh, isn't just about every other format in the game (and described by the IESDP) with the possible exception of BMP, WAV and maybe a couple others?

ACM and MVE formats are patent-covered (I.E. BIS/Bioware/Interplay/EA/Whomever will have a stronger case if they sue you).

[Miloch]

yep, reading the header will do (though I don't understand why Kivan is shipped with decompressed .wavs in the first place). I'm not sure about arrays, I haven't got to mess much with them.

This seemed to work. It did choke on one of Keiria's soundclips from DSotSC during acm conversion, but I resampled it in Audacity and it was fine after that. (For some reason, someone had redone all the DS clips as stereo and cranked up the decibels past the legal limit, but that's another issue...)

ACTION_DEFINE_ARRAY snx BEGIN ~ajant27~ ~ajant28~ ~ajant36~ ~ajant37~ ~alora35~ ~alora36~ ~alora37~ ~bedwin18~ ~bedwin35~ ~bedwin36~ ~bgmnsc29~ ~bgmnsc35~ ~bgmnsc36~ ~bgmnsc37~ ~bimoen06~ ~bimoen10~ ~branw29~ ~branw30~ ~branw35~ ~branw36~ ~branw37~ ~coran30~ ~coran35~ ~coran36~ ~coran37~ ~dynah25~ ~dynah35~ ~dynah37~ ~eldot17~ ~eldot35~ ~eldot36~ ~eldot37~ ~faldn25~ ~faldn35~ ~faldn36~ ~faldn37~ ~garrk23~ ~garrk35~ ~garrk37~ ~jaher30~ ~jaher35~ ~jaher36~ ~jaher37~ ~kagan24~ ~kagan35~ ~kagan37~ ~khald30~ ~khald31~ ~khald35~ ~khald37~ ~kivan35~ ~kivan36~ ~kivan37~ ~montr35~ ~montr36~ ~montr37~ ~quayl25~ ~quayl35~ ~quayl36~ ~quayl37~ ~safan35~ ~safan36~ ~safan37~ ~shart24~ ~shart25~ ~shart35~ ~shart36~ ~shart37~ ~skiee35~ ~skiee36~ ~skiee37~ ~tiaxx23~ ~tiaxx31~ ~tiaxx35~ ~vicon31~ ~vicon35~ ~vicon36~ ~xannn35~ ~xannn36~ ~xannn37~ ~yeslk28~ ~yeslk31~ ~yeslk36~ ~yeslk37~ ~concho27~ ~concho28~ ~concho31~ ~concho35~ ~concho37~ ~cuch24~ ~ferthg26~ ~ferthg29~ ~ferthg30~ ~ferthg31~ ~ferthg35~ ~ferthg37~ ~jetla04~ ~jetla22~ ~jetla27~ ~jetla35~ ~jetla36~ ~jetla40~ ~keiri23~ ~keiri24~ ~keiri26~ ~keiri27~ ~keiri37~ ~skezr13~ ~skezr23~ ~skezr25~ END

//WAVC sounds we need to adjust (the _wb is for mass deleting remains at the end)
ACTION_PHP_EACH snx AS dx => snd BEGIN
 ACTION_IF FILE_EXISTS_IN_GAME ~%snd%.wav~ THEN BEGIN
COPY_EXISTING ~%snd%.wav~ ~override~
  READ_ASCII 0x0 sgn (4) //File signature
BUT_ONLY
ACTION_IF (~%sgn%~ STRING_EQUAL_CASE ~WAVC~) THEN BEGIN
  COPY_EXISTING ~%snd%.wav~ ~override/%snd%.wavc_wb~
END ELSE ACTION_IF (~%sgn%~ STRING_EQUAL_CASE ~RIFF~) THEN BEGIN
  COPY_EXISTING ~%snd%.wav~ ~override/%snd%.wav_wb~
END
 END
END

ACM and MVE formats are patent-covered (I.E. BIS/Bioware/Interplay/EA/Whomever will have a stronger case if they sue you).

I don't doubt you (at least not for MVEs) but is the ACM patent documented somewhere? My understanding was that it's just an outdated Windows codec. Though that doesn't mean it's not patented necessarily, and maybe that refers to something else. If it is patented, I'd be curious as to who owns the rights.

There are even reports of area ambients needing decompression to standard .wav files (I can't remember the full discussion).

 

If I understand what you are saying enough, it seems like you could simply batch compress any preexisting raw .wav in override - the bigg has code (and the processing .exe needed) in Amber (certainly in the linux distribution) where he does this for all of her sounds.

Well, I don't know if I want to batch compress every single WAV, just the ones I'm dealing with if they aren't already compressed. There'd be even less justification for compressing all WAVs if your statement about ambients is true (I've no idea - not sure why it would be though since the ambients I looked at were all WAVCs).

 

It does make sense for modders to extract to WAVC instead of WAV, though without better (hopefully WeiDU-enabled) knowledge of the intermediary formats (assuming they're "legal"), the conversion process is rather tedious. But it'd certainly keep the override size down at least.

 

What sucks about using DOS batch files for all this conversion too is that, unlike WeiDU, it doesn't log the changes, and if it chokes on something, you might be kind of hosed as far as trying to uninstall what it did up to that point.

 

Edit: Interestingly, even WAVs are based on a file format introduced by EA in '85. So if EA owns (and enforces) a patent on the ACM format, that would be pretty interesting. I'm no lawyer, but I suspect it would be about as enforceable as the supposedly patented CompuServe GIF format, which is used just about everywhere on the web these days.

[Bearwere]

I am a little ways behind you guys - but if you are talking about things not being in WAVC format in he override, there are very few mods which use WAVC. Almost all use oggdec to compress to .ogg, and then best practice these days is to decompress to .wav in the mod folders and move uncompressed .wav to override.

I understand this logic for a new NPC, but (in Kivan case) taking existing .wavc, decompressing them to wav, compressing to .ogg, and finally decompressing to .wav is something I don't really get. What's the point?

 

ACM and MVE formats are patent-covered (I.E. BIS/Bioware/Interplay/EA/Whomever will have a stronger case if they sue you).

wow, I'm sooooo afraid (yawn)

 

I don't doubt you (at least not for MVEs) but is the ACM patent documented somewhere? My understanding was that it's just an outdated Windows codec. Though that doesn't mean it's not patented necessarily, and maybe that refers to something else. If it is patented, I'd be curious as to who owns the rights.

I think Interplay's .acm has but common extension with Microsoft's (though I'm not sure, nor really care)

[cmorgan]

Not sure about Kivan's case - I thought they were redone soundclips and included new voicing. I'll go take a look (later, like in a few months ).

 

I did see the bigg's code, which decodes the .ogg to .wav, then

 

wine snd2acm.exe -wav mxm\#ama.wav mxm\#ama.acm

 

 

[offtopic]

If I were to do a trial run for fun with everything in the override compressed, would it be (windows [DOS]

 

install

@echo off
test/snd2acm -wav  override/*.wav
move override/*.wav test/backup/wav

 

uninstall

@echo off
move test/backup/wav  override
del override/*.acm

 

The 'move ambients to override' problem was (I think) associated with the beregost crash bug, which makes any speculation awkward. I'd like to see if there is a way to get compression going and reduce the "bif the override" idea, since it kinda defeats WeiDU completely...

[/offtopic]

[Miloch]

If I were to do a trial run for fun with everything in the override compressed, would it be (windows [DOS]

install

@echo off
test/snd2acm -wav  override/*.wav
move override/*.wav test/backup/wav

I don't think snd2acm takes wildcards, so you'd have to resort to hackery. Also, you'd have to convert the resulting acm to wavc by adding a header.

BACKUP ~1wavtest/backup~
AUTHOR ~forums.gibberlings3.net/index.php?showtopic=15178~
VERSION ~Alpha 1~

BEGIN ~WAV to WAVC Test~

<<<<<<<< .../1wavtest-inlined/wavcinst.bat
>>>>>>>>

COPY + ~.../1wavtest-inlined/wavcinst.bat~ ~1wavtest~
COPY + ~.../1wavtest-inlined/wavcinst.bat~ ~1wavtest/wavcmid.bat~
COPY + ~.../1wavtest-inlined/wavcinst.bat~ ~1wavtest/wavcutmp.bat~

COPY_EXISTING_REGEXP GLOB ~.*\.WAV$~ ~override~
 READ_ASCII 0x0 sgn (4) //File signature
 PATCH_IF (~%sgn%~ STRING_EQUAL_CASE ~RIFF~) BEGIN
SPRINT $df(~%SOURCE_RES%~) ~%SOURCE_RES%~
INNER_ACTION BEGIN
  APPEND_OUTER ~1wavtest/wavcinst.bat~ ~"1wavtest\snd2acm.exe" "override\%SOURCE_FILE%" "override\%SOURCE_RES%.acm"~
  APPEND_OUTER ~1wavtest/wavcinst.bat~ ~move "override\%SOURCE_FILE%" "1wavtest\backup"~
  APPEND_OUTER ~1wavtest/wavcutmp.bat~ ~move "1wavtest\backup\%SOURCE_FILE%" "override"~
  APPEND_OUTER ~1wavtest/wavcmid.bat~  ~del "override\%SOURCE_RES%.acm"~
END
SET EVALUATE_BUFFER ~%SOURCE_RES%wavsize~ = SOURCE_SIZE
 END
BUT_ONLY

ACTION_PHP_EACH df AS i => xs BEGIN
 COPY_EXISTING ~%xs%.wav~ ~override~
END

AT_NOW ~1wavtest/wavcinst.bat~

ACTION_PHP_EACH df AS i => xs BEGIN
 COPY_EXISTING ~%xs%.acm~ ~override/%xs%.wav~
acmsize = SOURCE_SIZE
INSERT_BYTES 0x0 28 //ACM header is already attached
WRITE_ASCII 0x0 ~WAVCV1.0~ #8
WRITE_LONG 0x8 EVALUATE_BUFFER ~%SOURCE_RES%wavsize~ //Uncompressed
WRITE_LONG 0xc acmsize //Compressed
WRITE_LONG 0x10 0x0000001c //1st part of WAVC constant
WRITE_LONG 0x14 0x00100001 //2nd part of WAVC constant
WRITE_LONG 0x18 0x77e75622 //3rd part of WAVC constant
 BUT_ONLY
END

AT_NOW ~1wavtest/wavcmid.bat~

COPY + ~1wavtest/wavcutmp.bat~ ~1wavtest/wavcunin.bat~

AT_UNINSTALL ~1wavtest/wavcunin.bat~

It works on minimally modded Tutu at least, and even produces audible WAVCs that don't sound like Donald Duck quacking into a paper bag after inhaling from a helium balloon (the few I listened to anyway). But I'm not sure how smart it is or whether it really benefits an install at all. For one thing, it'll extract uncompressed WAVs from game BIFFs (there are a few) as well as deal with those already in the override. I'm not sure how to get around this, since I couldn't find a sort of COPY_EXISTING_REGEXP GLOB_ONLY or COPY_REGEXP type of syntax that *only* looks in the override and not BIFFs. There probably is one, I just didn't see it. Secondly, it leaves any uncompressed (originally extracted) WAVs in the override on uninstall, which kind of defeats the purpose of reducing the override size. But if I deleted them, I might be deleting unbiffed WAVs added by a mod, which would not be good. It does delete the other junk files produced by the batch files though. There's probably a better way of coding this - I didn't spend a whole lotta time on it. WeiDU was doing bizarre things with SOURCE_FILE misrepresentation as mentioned recently on PPG, so I had to lean on ACTION_PHP_EACH instead.

[cmorgan]

Bleh.

 

Not your hard work, that is appreciated - it is just that you are right. Under those conditions it does not accomplish what I wanted to try.

 

I was hoping for a way of skipping WeiDU and simply using OS/command line to convert the .wav directly to the wavc format - thinking wavc would work just as well in the override as a wav.

 

If it involves messing with headers, etc., then (unfortunately) creating a .bif of sounds is simpler. Not a good option, and it again defeats WeiDu's basic premise, but certainly better than that for my purposes.

[Miloch]

I was hoping for a way of skipping WeiDU and simply using OS/command line to convert the .wav directly to the wavc format - thinking wavc would work just as well in the override as a wav.

 

If it involves messing with headers, etc., then (unfortunately) creating a .bif of sounds is simpler.

It does what you wanted by producing the DOS batch files dynamically with APPEND_OUTER and running them via AT_NOW. The last block attaches the 28-byte WAVC header, which is pretty standard. Undoubtedly it would result in a smaller override folder if you had even one voiced NPC installed with uncompressed WAVs in the override. It might even result in smaller biffs if you run it before biffing - not sure how biff compression works. There must be some benefit to biffing WAVCs, or they wouldn't have bothered making them in the first place (unless the intent was just to obfuscate the audio, which would be silly since any recording program could reproduce it).

[cmorgan]

I will give it a shot then, and see what happens - I am very interested in seeing if this could reduce the load of modern voiced npcs. Adding specific instructions to the combined audio installer might be the way to go.

[Miloch]

This seemed to work. It did choke on one of Keiria's soundclips from DSotSC during acm conversion, but I resampled it in Audacity and it was fine after that.

Well, although I had the Frame Rate component working on my BGT/BWP install, it kept bombing out on erebusant's, so we decided to shelve it for now. Kept throwing errors like this in the debug log:

ERROR: [./override/branw35.acm_wb] has size 28901780: TOO BIG FOR WEIDU (max 16777211)

ERROR: [./override/eldot17.acm_wb] has size 28888984: TOO BIG FOR WEIDU (max 16777211)

ERROR: [./override/kagan37.acm_wb] has size 28901167: TOO BIG FOR WEIDU (max 16777211)

No idea why it would be generating 28+ MB (or whatever) sound clips that are only a few words long. My suspicion is that the snd2acm code isn't exactly up to par (it is a beta 1 after all). And my hope is that someday we can recode this whole conversion process in WeiDU alone, using the DLTCEP C++ code as a basis, but that's probably a remote dream for now, at least for my coding skill level.

 

But yeah, it does make sense that mod authors should be decoding to WAVC instead of plain WAVs. I wonder what the difference would be between a RAR on best compression with OGGs and the same RAR with WAVCs. If there's not much difference, we could dispense with the whole oggdec thing.

[cmorgan]

That would be cool, I think - but given legacy mods, I am just hoping to be able to steal someone's code to auto-compress the suckers to wavc on the fly.

[jcompton]

(side note - players report installs with a ton of NPCs with soundfiles cause lag. I wonder if the engine would be happier with WAVC format instead? I can't imagine everyone has been putting raw .wav files in place when the WAVC format has been available and would reduce the size of the override, unless someone had determined that .wav was better in some way - may have to go thread-archeology-searching, or beg Smoketest and devSin to weigh in...)

 

I strongly suspect that override lag is a function of number of files in override, not number of kilobytes in override. Searching for game resources in override is presumably much less efficient than searching the optimized chitin.key index. Unless the game is ridiculously stupid, they're just looking at the filenames--not searching within the files in override.

 

So WAVC vs. WAV will get you nowhere from a performance standpoint.

 

I wonder what the difference would be between a RAR on best compression with OGGs and the same RAR with WAVCs. If there's not much difference, we could dispense with the whole oggdec thing.

 

While I understand the urge to not simply take conventional wisdom at face value, do you genuinely think nobody's thought of this before? Like, I forgot to do the math before releasing Kelsey and nobody considered challenging me in the 6+ years since?

 

Give-or-take, WAVC compressing a WAV is a 2x improvement.

 

Give-or-take, RARing a WAV is a 2x improvement.

 

RARing a WAVC saves you next to nothing--so you're still at a 2x improvement.

 

Ogging gets you about an 8x improvement.

 

8 > 2.

 

There must be some benefit to biffing WAVCs, or they wouldn't have bothered making them in the first place (unless the intent was just to obfuscate the audio, which would be silly since any recording program could reproduce it).

 

Interplay loved using WAVC whenever possible and, overall, it probably did save them an extra CD. Again, a biff isn't really about filesizes, it's about numbers of files. Looking through one big file (especially when you have an index) is easier for a filesystem to do than to look through thousands of files for a specific one.

[cmorgan]

Cool info - and thanks for weighing in.

 

The problem isn't that we don't think you hadn't weighed it all, it was that we (I) hadn't been able to find evidence in searches of past forum posts - and an in-depth read of early threads wasn't on my docket yet. Creative brainstorming, and all that stuff.

 

So it sound like the real answer to the question *I * am looking at (not Miloch's wavc fixes to sounds) is

 

1. The community (or at least JCompton, and folks followed his lead) looked at this years ago, and figured out that there was no significant difference in size between using [raw .wav ogg'd and decompressed to raw wav] and [wavc ogg'd and decompressed to wavc].

2. The engine slowing under a heavy soundfile NPC load is probably a function of how many files there are in the override rather than the size of those files.

3. While the engine reads wavc and wav files, and converting existing files in override from wav to wavc would reduce the added file size by about 50%, this does not achieve anything for either modder or player. (There is no community-wide attempt to limit the size of the installed mod other than Macready's EasyTutu options to cannibalize the other games - the concern about size is download speed for slow connections).

4. Since the .wav files work fine and there is no more or less lag introduced by converting to WAVC, it is a PITA to the modder to take an extra step which will result in no percieved benefit.

 

So, basically, the most efficient way to reduce lag is the indexing of those files via .bif (which really destroys the whole install/uninstall thing, but that's the way it is). Looking for ways of reducing filesize to reduce lag is a red herring.