Game(s) Database?

[Mike1072]

Here is what I mean.

 

"-" = represents a field within a file/database from the base game.

"a" = represents a field within a file/database from the ModA

"b" = represents a field within a file/database from the ModB

 

...

 

Another case would be:

 

----------- (10 fields within the database, 0 thru 9)

----------a (would change field 9)

---------b- (would change field 8)

 

Now, there are a number of possibilities of what the developer wants, he wants value b in the in field 8 field, but he wants the base game value for field 9 or i he may want the value b and whatever field 9 is currently or he may want the value of field 10 to always be set as per ModA (or a list of Mods).

 

This just describes a SET operation, ADD for instance, would allow the field to be added to:

 

 

----------- (10 fields within the database, 0 thru 9)

----------+N (would add N to field 9)

----------+N (would add N field 9)

I don't think you would want to incorporate mod installation logic into the database. That would require a drastic overhaul of how mods are written and it would be equivalent to installing the entire set of mods each time the game is run. And for what benefit?

 

Realistically, after a mod was installed, you could see which files it modified and store the modified copies.

 

 

So, the information you could store from your first example would be:

----------- (10 fields within the database, 0 thru 9)

----------a (would change field 9 to a)

---------ba (would change field 8 to b)

 

And the second example:

----------- (10 fields within the database, 0 thru 9)

----------n (would change field 9 to n, equivalent to whatever - plus N is)

----------m (would change field 9 to m)

 

 

The problem is that it's entirely possible that a mod will have a different effect depending on whether another mod has been installed or not. Even if you could do some sort of diff and revert the changes from one particular component, you wouldn't necessarily get the same installation that you would if you had never installed that component in the first place.

 

This can be seen from your second example, because you wouldn't really know if m was calculated by adding N, or throwing in an m no matter what was there previously, or even throwing in an m if it was n previously but throwing in a q if it was something else, etc. The same is true for your first example: you don't know which scenario you described (if any) accurately represents the second mod's logic, so the only way to replicate its behaviour is by actually installing it.

 

Here is a real example. Many mods utilize the custom scripting state CD_STATE_NOT_VALID. This a state that doesn't exist in the base game, but it is useful, so many NPC and quest mods add it to the game file state.ids. The way it is added is important: they read state.ids to make sure it's not already there before they add it. This prevents the file from filling up with hundreds of the same entry.

 

If you have 10 mods that use CD_STATE_NOT_VALID, they will all have code to add it to state.ids, but only the first mod installed will actually execute it. As far as you can tell from examining files after each mod is installed, the first mod is the only one that changes that file. If you try to disable that mod by plucking out its changes without re-running the code that installs the others, CD_STATE_NOT_VALID will be removed from state.ids and the other mods will stop functioning properly.

[Avenger]

1. Fork GemRB.

2. Implement database.

3. - (well, i have my doubts)

4. Profit. (if it works, good!)

[lynx]

Another real benefit is game saving.

 

You could have a checkpoint a bit like the Auto-Save file, but instead of saving a snapshot of how the world looks now, it would just save the differences (delta's). This means that you could easily go back in time to a particular date/time! Since only the delta's are saved, the amount of data would be trivial, so you could easily store 100+ days back in one hour GT increments. Also it would be possible to merge just the changes from a particular time back into the current time, again auto-magically resolving any conflicts, and having gone back in time, you decided you liked it the first time around, you can easily go back!

Like in real life, you can't time travel and expect the present to behave as expected. There is too much logic that is not purely data-driven for you to be able to just swap table contents. Also, a bit less important, but you can't save at all times, so the choice would always be limited.

 

In GemRB's case, for a similar effect, it would be easiest to make a mod that uses iwd2-style autosaves and then just add them throughout the game.

 

But you're the first that is interested in extra save games every five minutes.

 

You wouldn't really need to provide calls back for change notifications (although there are provisions for it if you want to be notified on a field-by-field basis) as such, the data would be refreshed into the in-memory object by the Engine and the common (base class) of the object in question. They'd be an end transaction status call back that could probably be handed generically for most cases and where it needed to be handled you could override this and provide you own, possibly calling the base implementation.

 

Anyway, I'm going to finish doing the first part of building of the database, because I've started it and I want to use it for my own editor for BG-EE, BG2-EE Mac.

That or normal functions, which you for some reason dislike. Really, there is much more to it than just storing object state, especially when it comes to effects.

 

A really good example of what I am talking about here is that fact that there are two almost identical file structures for the BG-EE and BG2-EE games. This is because the game was originally sold only on the Beamdog/Steam/Whatever. Then they made an Mac Store version which requires a different file structure, so, a lot, if not all of the mods and their instructions stop working! There have been some changes to the installers to fix this but it's still a problem, especially on iOS where the file system is hidden and difficult to get at by the user. What I am saying is that if all this information was stored in a single file with a clearly defined interface all these problems would disappear. Engine Developers and Mod developers alike could forget about the file system and the packaging and get on with the content!

 

If they suddenly change the way in which the file system is accessed on either iOS or Android (as has been the case many times on iOS), then what? You'd have to recode and possible make different versions to cope with it.

 

I'm an Apple developer and I know for a fact that file paths will soon all become URLs and URIs (to the outside world) which will break everything again I should imagine!

Only the game developers need to care about that. From the modder's point of view, all the details are contained in chitin.key and the actual layout has little effect.

[LookToWindward]

It depends how you set up the table/field in the database, fields can be Unique (e.g. there is only one of them) or they can be duped, in this case it would obviously be defined as Unique. So each mod would try to add it, if it is already ready there, then the add is ignored, if it's not there it is added.

 

I think we are talking at cross purposes, if you could supply a simple real life example of:

 

ModA Changing a something in a file.

ModB Changing the same thing in a file.

 

The way is it currently, there must be Mods that won't work correctly if there are installed with/without another mod being already installed.

 

There are problems in the way Mods are installed with the current system,. as you describe above (e.g. conflicts) which can maybe be resolved by the Mod installation process or maybe by some form of human intervention or maybe not at all. Whatever happens currently would happen using the database unless some information is added to aid the installation process.

 

However, having a database, would allow this to be achieved much more easily and in a more consistent and controller manner.

 

Anyway, as I said there are two stages to this, firstly it's importing all the existing files into tables into a database. I'm about 2/3 of the way through doing this now. When this is complete, there will be the whole of the data from BG-EE will exist in one structured file. I will then add the file exporters.

 

At this point I will have enough to get on with my editor.

 

Stage 2 would be to develop a Model/Schema that handles all the other things we have been discussing. I don't much care if this is done or not, I think it would make a lot of sense and would be good in general for games like BG and BG-EE but if there is no interest then it doesn't really matter.

 

At some point though in the not too distant future though, especially on the mobile side, the current way of doing things will break and there will be no easy fix for it. The mobile OS Companies have been sending a message to developers for a while now, which is basically App's need to be sandboxed, this is why there are problems with the Mac Store version of BG, on the Mac, at the moment, you can just about get around the problem, but for instance, this involves changing file/folder permissions which is not really advisable since it can lead to security holes.

 

Basically the whole process is dependant on the OS Filing System and while that is not a bad dependancy for desktop machines (although recent OS updates and the Mac App Store have made to more difficult), on a mobile device its really not a good idea.

 

---------------------------

 

From re-reading some of the above, I think people are still thinking of File X, containing information Y, rather than forgetting about how the data is packaged and the internal ordering of things. Once all the files in a Game-Set have been imported into the database, there will no concept of a File X, there are just fields. A database is a bit like the Borg in that it assimilates all!

 

So you can treat your data as one big lump and anything that affects something else in the database is automatically handled using rules from the Schema.

 

The major pit-fall of this approach is that *everything* is dependant of the Database Engine API and the underlying code. A bug in the DBE would affect EVERYTHING! This sounds worse than it actually is but it does mean that the Engine, especially the Schema needs careful design and a LOT of testing.

[LookToWindward]

But you're the first that is interested in extra save games every five minutes.

 

I'm not really interested in that, but something akin to it might be a good feature. For instance, I'd love to be able to track my movements over the last (say) 24 hours game time. There are plenty of times, after loading a saved game after a few days of not playing, that I've forgotten exactly where I've visited and it takes me a while to re-orient (this is especially true if like me you play two or more different games at the same time), I would really love a facility to show my movement up to where I am now. Something like:

 

Day 100, Hour 12 Current Time

 

Hour 12 - Nashkel, <name of Inn - I've forgotten!)

Hour 11 - Nashkel, Person X's House.

Hour 10 - Friendly Arms Inn.

Hour 9 - NPC X Joined Party.

 

You could easily extend this, for instance, to show the Inventory items held at each point.

 

I'm not suggesting that this can't achieved in another way or that it's even wanted by anyone else (although I've seen Feature Requests for it for BG-EE), but it would take knife-and-fork code to do it, whereas you get a very efficient/robust way of doing this almost for free as a side-effect of using a database.

[lynx]

1 ingame hour == 5 real minutes.

 

I see we will not reach any sensible conclusion here, so like Avenger hinted, just convice us with the code.

[LookToWindward]

Yes, that's what I was thinking too.

 

Don't take the numbesr too literally, you'd obviously choose sensible numbers, but two points come from this:

 

1. if you did it every 5 Minutes real time, there would be very little difference, if any between the current and last state most of the time, so nothing would be saved. If however, you had a monster haul, you could easily pick up 100 items in 5 mins, in which case it would be noted.

 

2. Currently, when you save game, it makes a full copy, so if you did save it every 5 mins real time, you'd soon run out of storage, whereas with the database approach if you saved a game and nothing had changed expect maybe the time, then the time is all that would be saved.

 

This is the fundamental difference between the two ways of doing things.

 

As soon as I get this working I'll send you the database and you can take a look at stage 1.

 

As for Stage 2, if anyone thinks it worth it, I'd probably need help some help to figure the Schema.

[lynx]

Do work on your editor, as you left us unconvinced about the rest.

 

Obviously nobody would pick 5 minutes, as that means so many saves you'd have to create a very good loading and saving gui for the autosaves to be usable. On a modded game, the save size averages around 1M here, so even with that you could easily finish the game without filling 1G. But it doesn't really matter.

[Mike1072]

I think we are talking at cross purposes

You may be right. I'm all for making modding easier, so I've spent some time considering alternatives to the way we mod currently. Let me try to elaborate on my concerns.

 

From re-reading some of the above, I think people are still thinking of File X, containing information Y, rather than forgetting about how the data is packaged and the internal ordering of things. Once all the files in a Game-Set have been imported into the database, there will no concept of a File X, there are just fields. A database is a bit like the Borg in that it assimilates all!

All existing mods deal with files. They read current game data from files to make decisions about how to modify files.

1. What mod data are you planning to store in the database?
   


1. The changes to the game data as a result of installing the mod (e.g. diff showing CD_STATE_NOT_VALID was added to States)
   
2. The logic for modifying game data as would be used when installing the mod (e.g. "Add ~CD_STATE_NOT_VALID~ to States if not present")
   
3. Something else
   

[*]Do you intend to support mods that were not designed to interact with your data model?

[*]If so, how are you planning to accomplish this?

1. Allow mods to install on the file system as normal, then store the changes they have made
   
2. Something else
   

 

1a/3a would be straightforward to develop, but I don't see how that could lead to a system where mods could be enabled or disabled at will.

 

1b would allow for enabling or disabling mods, but it would be quite cumbersome to develop, especially if you want to support existing mods or mods of a similar complexity.

[LookToWindward]

You are seeing "saves" as a separate "file" thing, the point is, if you set it up right, it would save whenever there is an (appreciable, programmable) difference between the state over time, and, it would only "save" the differences. IOW, there is no explicit "save" operation performed, it comes as a by-product of the the way the engine works. It does everything necessary for a "save" anyway, it just that the results of the "save" are not explicitly stored anywhere unless you want them to be.

 

The database building part is quite easy and quick, the GUI for the editor will take a bit longer, but as soon as I have something, I'll share it.

[LookToWindward]

All existing mods deal with files. They read current game data from files to make decisions about how to modify files.

 

The way I see it working initially, is that the first time the game is run (or if a re-build is performed), each file in imported in a hierarchy of tables (classes/objects). By each file, I mean the key file is read and an entry is created in the master game table for that game. The key file is then read and each resource type in the files it points to (e.g. rippling through, BIFF files etc). At the end of this process, the database would contain a hierarchy of tables containing the data. This creates basically an indexible store of all the content based on keys. Although each individual table would serve a similar function as the real filing-system file, they are now part of a game-set. You can then write stuff on top that treats each individual field across all "files" in the database. For instance for BG1-EE:

 

There would be a table called something like "BG1EEGame", this would contain and "BaseGame" section that references the other lower level objects (files). From this perspective, it no longer matters that the creature "GNOLL" originated from the "creatures.bif" file, it would just be part of a lump of data that is part of "BG1EEGame" to the caller.

 

Now say you added "ModA" to the the game "BG1EEGame". It would add "ModA" as a sibling of super/parent class/table of "BG1EEGame", IOW, it would perform the same import operation as for the main game, but it would create it own set of data, with ONLY the differences. This would mean that if you wanted to override "GNOLL" and, say, make it's Hit Points 50 for the rest of the game, then this one Hit Points field would be different, if the "ModA" table, all the other fields, such as strength, dexterity, would be "NULL", meaning get it from the parent,

 

It's really hard to explain in text, but at the low level, the tables might look like this:

 

BG1EEGame->BaseGame->Creatures->GNOLL->STRENGTH = 16 --Original values from initial base game import

 

BG1EEGame->BaseGame->Creatures->GNOLL->DEXTERITY = 18 --Original values from initial base game import

 

BG1EEGame->BaseGameCreatures->GNOLL->HP = 10 --Original values from initial base game import

 

BG1EEGame->ModA->Creatures>GNOLL->HP = 50 --Override HP from initial base game import for ModA.

 

When these three fields are read, STRENGTH and DEXTERITY will be read from he

 

1 What mod data are you planning to store in the database?

 

Basically the same as in the files now, for Stage 1, for Stage 2, you'd design schema that defined the relationships etc, IOW, which fields relate to which other fields. All the different fields from all the files imported are available as if they were individual items of data, so if, for instance you wanted the Player Name (from the CHR file) and the HP (from the CRE file), you would just ask for these fields not caring that that are in separate files (maybe via a BIF file).

 

a The changes to the game data as a result of installing the mod (e.g. diff showing CD_STATE_NOT_VALID was added to States)

b The logic for modifying game data as would be used when installing the mod (e.g. "Add ~CD_STATE_NOT_VALID~ to States if not present")

c Something else

 

As above, if "GNOLL" didn't exist in the base game it would be added. It really boils down to what the Mod developer wants. Right now, it would do whatever the non-database version would do, but for instance, when a Mod Developer adds/changes something they would have the choice of:

 

Create (must not already exist).

Add (will add only if not already there).

Update (will update the field)

UpdateOrAdd, will update the field if it exists, otherwise it will add it.

 

There would be different scopes too, for instance you could specify the above to only look in BaseGame, or ModA or whatever.

 

2 Do you intend to support mods that were not designed to interact with your data model?

 

Yes, they will be imported in the database at install time. I think that once people got to trust this way of doing things they would just ship an SQLite file for the mods. This would be better, but if not, then creating them from the raw files does the same thing, it's just it will take a bit longer, mainly because the indices/tables will have already been built in an SQLite file, whereas they need to be built when importing.

 

3 If so, how are you planning to accomplish this?

 

As above, import the raw files into sensible tables, which is enough for my Editor, then design a schema above it to link the relationships and the rules.

 

3a Allow mods to install on the file system as normal, then store the changes they have made

 

Yes, it would work, but better/easier to just point the importer at the Mod Folder and it would merge the two in the database. If you wanted to copy the files into the game folder that would work too, it's just it would take longer.

[LookToWindward]

I've got no clue why my last post is full of html crap - such is life! Browsers make such bad editors. sigh........

[Mike1072]

I've got no clue why my last post is full of html crap - such is life! Browsers make such bad editors. sigh........

Yeah, that seems to happen here randomly. I tried fixing up your post so I can actually read it. I'll respond later.

[lynx]

we're waiting, Mike.

[Mike1072]

we're waiting, Mike.

Pssh, I need my beauty sleep.

 

2 Do you intend to support mods that were not designed to interact with your data model?

 

Yes, they will be imported in the database at install time.

 

The problem with this strategy is that it won't let you freely toggle mods off and on, if you only store the changes from installing a mod on one particular set of game data*. You don't know what effect the mod would have if it were installed on a different set of game data.

 

I'll refer back to the CD_STATE_NOT_VALID example. The base game's state.ids file has no CD_STATE_NOT_VALID, so mods that use it add it when another mod hasn't done so already.

 

Say you have two mods that use CD_STATE_NOT_VALID, Mod A and Mod B, and you are going to install them on a clean base game.

 

First, you install Mod A. Importing its changes to the database will store a change to state.ids (it adds CD_STATE_NOT_VALID because it's not there).

Then, you install Mod B. Importing its changes will not store a change to state.ids (it doesn't add CD_STATE_NOT_VALID because it's already there).

 

Now, if you were to disable Mod A in the database, state.ids would be reverted to its original condition, and if you were to play the game, Mod B's content would not work.

 

This is a very simple example. Mods often take special account of one another to interact with their custom content or adhere to altered rules. The only way to know what the effect of installing Mod B on a specific set of game data is to perform that installation.

 

 

 

*You could install the mod on ALL sets of game data, but that would rapidly spiral out of control as I mentioned here.