diff tool for IE formats?

[critto]

WeiDU has --cmp-from and --cmp-to which allows to see a raw sequence of write_byte instructions to convert one version of the file into another. Did somebody ever go one step further and prepared a tool that analyses this output and prepares a patch in a more readable, modder-friendly form? For example, if I am preparing a patch for a creature file based on this information, a sequence of write_bytes for a 0x0248 offset (which is the creature script) would be converted into a more appropriate WRITE_ASCII and so on?

 

I have lots of modified files that were changed manually in a GUI editor (the old-school approach) and then copied straight into the override folder via a simple copy. It's not very flexible. I want to see in greater detail what kinds of changes exactly were applied to those files and, theoretically, convert all hard copies into proper patch sequences. The question is whether I will be spared the challenge of writing a utility like that myself.

[lynx]

I did write a simple wrapper, but the dumpers (ieparse or ielister) all produce human readable output, so at best it could be an aid(e) for you.

[critto]

Those are both parts of GemRB, right?

[lynx]

No, one is part of iesh (where also iediff is), the other separate like dltcep:
https://sourceforge.net/p/gemrb/ie_shell/ci/master/tree/
https://sourceforge.net/p/gemrb/ielister/ci/master/tree/

from the dates you can see that both are incomplete as far as file formats go, missing a few updates. For what we need it in gemrb (plaintext diff for git log), ielister works better, but with iesh you get a full shell, so it's great for exploration (eg. which areas have this bit set? how many weapons with ranged headers are there?).

Example output:

commit 571100d62479082b9f7a4b52abae94384928e540
Author: Jaka Kranjc <lynxlupodian@users.sourceforge.net>
Date:   Fri Sep 18 08:26:58 2015 +0200

    iwd2: fixed a few recent clab names

diff --git a/gemrb/unhardcoded/iwd2/39280.spl b/gemrb/unhardcoded/iwd2/39280.spl
index f26a0a5..74ae130 100644
--- a/gemrb/unhardcoded/iwd2/39280.spl
+++ b/gemrb/unhardcoded/iwd2/39280.spl
@@ -1,5 +1,5 @@
 00h signature    spl v2.0
-08h unid name    0000998f
+08h unid name    00009970
 0ch id name      0098967f
 10h completion           
 18h attributes   00000000

commit bc395d1c839396fba28588223101be2c02326198
Author: Jaka Kranjc <lynxlupodian@users.sourceforge.net>
Date: Mon Jul 20 18:59:03 2015 +0200

iwd2: wizard variant of confusion targets everyone

diff --git a/gemrb/unhardcoded/iwd2/confusw.pro b/gemrb/unhardcoded/iwd2/confusw.pro
index ea6e585..b1912a9 100644
--- a/gemrb/unhardcoded/iwd2/confusw.pro
+++ b/gemrb/unhardcoded/iwd2/confusw.pro
@@ -137,7 +137,7 @@
01f8h unknown 00000000
01fch unknown 00000000
0200h aoe section
- 0200h aoe flags 000004c0
+ 0200h aoe flags 00000400
0204h trigger radi 012c
0206h effect radiu 012c
0208h explosion so are_m21

 

 

[critto]

Thanks for the suggestion. I've set it up, but the intial load_game() fails:

  File "/Library/Python/2.7/site-packages/infinity/stream.py", line 372, in seek
    self.fh.seek (offset)
AttributeError: 'NoneType' object has no attribute 'seek'

I suspect this is because iesh does not support the EE? A vanilla ToB game seem to load fine.

[lynx]

Could be, I've never tried it. Are you running it on some new formats?

[critto]

What do you mean by new formats? I am trying to load the game itself. I haven't gotten as far as browsing the game data.

[lynx]

ah, well then the answer is obvious. I don't think Ed has any of the EEs, so someone else would have to look at it.

[critto]

I can try to check it out if somebody gives me directions what to look for.

[lynx]

I'd do the usual, either start putting prints before that error line or a pdb call (if you're more familiar with python) for a simple debugging shell. Which file did it choke on?

[critto]

My guess is, it was the chitin.key

 

 

Cmd: load_game("/Users/critto/Documents/bg2ee")
Loading 186 file refs and 60118 RESREFs. This may take ages
.0..................................................5000..................................................10000..................................................15000..................................................20000..................................................25000..................................................30000..................................................35000..................................................40000..................................................45000..................................................50000..................................................55000..................................................60000.()
Traceback (most recent call last):
  File "/usr/local/bin/iesh", line 308, in <module>
    exec (current_command)
  File "<string>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/infinity/builtins.py", line 65, in load_game
    core.strrefs = stream.get_format () ()
  File "/Library/Python/2.7/site-packages/infinity/stream.py", line 266, in get_format
    signature = self.get_signature ()[:8]
  File "/Library/Python/2.7/site-packages/infinity/stream.py", line 235, in get_signature
    s = self.read_sized_string (0x0000, 16)
  File "/Library/Python/2.7/site-packages/infinity/stream.py", line 158, in read_sized_string
    self.seek (offset)
  File "/Library/Python/2.7/site-packages/infinity/stream.py", line 372, in seek
    self.fh.seek (offset)
AttributeError: 'NoneType' object has no attribute 'seek'

[lynx]

it looks more like something it references. fh is clearly None there and not the object (file handle) we expect. Why not — where is it usually set up ...

 

btw, for the differ, I think you don't need to run iesh or involve chitin.key, since you pass it the file paths directly.

[critto]

I think EE introduces some weird new files that aren't IE formats. There was a similar issue with SCS, I wrote about it a couple of days ago. I'll try to find what's the reason for failure.

[critto]

No, the problem was that iesh couldn't find dialog.tlk at those locations where it's expected to be by default. A None instead of a proper file handle is passed to the Stream object. If I provide a correct path, it loads OK.

Cmd: load_game("/Users/critto/Documents/bg2ee", "chitin.key", "lang/en_US/dialog.tlk")
Loading 186 file refs and 60118 RESREFs. This may take ages
.0..................................................5000..................................................10000..................................................15000..................................................20000..................................................25000..................................................30000..................................................35000..................................................40000..................................................45000..................................................50000..................................................55000..................................................60000.()
/Users/critto/Documents/bg2ee/lang/en_US/dialog.tlk
Loading 108629 STRREFs. This may take eternity
.0..................................................5000..................................................10000..................................................15000..................................................20000..................................................25000..................................................30000..................................................35000..................................................40000..................................................45000..................................................50000..................................................55000..................................................60000..................................................65000..................................................70000..................................................75000..................................................80000..................................................85000..................................................90000..................................................95000..................................................100000..................................................105000....................................

[critto]

Your iediff is seriously awesome It makes at least half the job done. Now I'll try to figure out how to write WeiDU patch generator based on the info it provides.