projectinfinity Delta Updates for mods hosted at GitHub - reliable, lightweight and fast updates for you mods

[AL|EN]

Hi,

One of the most missed features for all past years was the ability to click one button in order to update my all local mods. Lucky, with the GitHub API, Project infinity can offer a possibility to update mods, which is very easy to add via one single line inside mod metadata file. Because it's GitHub-based, it offers stability and consistency. It is impossible to download mod with malformed files, you either download it correctly or don't download it all (git feature). And for the first time, there is a possibility for so called "delta updates"!

As a sidenote: GitHub.com now offers private repositories for free!

Right now, this system is opt-in and it's based on the mod metadata "Download" keyword:

Download=https://github.com/Gibberlings3/SwordCoastStratagems

Features:

• easy to integrate
• simple and understandable rules for modders about when mod will be updated and how
• the update process don't touch mods inside game directory, only the mods from the directory where user has placed all mods

Requirements:

• Mod use GitHub hosting (or the site which provide API)
• Mod has metadata file and filled 'Download=' keyword

General Flow:

• Phase 1: Read Mod Metadata  "Download" keyword > Check if update exist > Present 'Update Available' for specific mod > do nothing until player hit "Update Mod" button
• Phase 2: the "Update Mod" button was clicked > Download update > extract and overwrite all mod folders and files / preform delta update and also overwrite all mod folders and files

How it would work:

• Checking if there is a updated version of the mod is done automatically (but not the update itself)
• Notification about the update will be presented to the player ('Update Available' next to the mod name, "Update mod" button enabled")
• The update itself can be only accepted/started by player himself (in order to not break existing mod/components setup)
• Leave user-created files inside mod data folder intact for eg: mymod\mymod-config-user.ini

Which update channels would be offered for players:

• if mod has releases, offer to update only if there is a new release
• if mod has only prereleases, offer the update for prereleases
• if mod has both releases and prereleases, offer the update for releases only, do not offer the update for prereleases unless there is a global option "Allow for prereleases" enabled
• if mod doesn't have any releases or prereleses, the assumption is made (mentioned at 'Download=' keyword documentation ) that the modder use master branch. For many modders, simply committing to master branch seems to be enough and I don't want' to force them to make releases. Modders can simply create Release in order to switch update chanel.

When player decides to update the mod, there are two ways of updating locally extracted mod files:

• downloading release/prerelease, extract and overwrite all mod folders and files
• preform delta update, also overwrite all mod folders and files

My main concerns are:

1. Should I offer "Prereleses" for normal users, who aren't interested testing BETA versions and reporting bugs? (opt-in option?)
2. Should I offer "download last commit" aka "experimental" for mods which do have releases/prereleases? (undocumented option?)
3. Should I offer "download last commit" from specified branch like "devel" (undocumented option?) @argent77 AFAIK, you alone use different branches.
4. What to do with files which were available in older version (let's say MYMod\Items\SWORD.itm) but the new one doesn't have them anymore? Removing old mod version directory entirely could potentially remove also user-generated config (only two mods have them but it's still important). If such problem occur, any kind of solution for is doable at this stage. No worries, files non-present inside new mod version are removed.

That's everything for now, any feedback will be appreciated.

@K4thos Enabling this for EET will allow for EET delta update without re-downloading 250+ MB, what do you think?

Edited December 31, 2020 by AL|EN
cosmetics

[lynx]

1. Unless SR and IR (iirc) got new releases, they still use both mechanisms for two different release streams (3 and 4).

2. and 3. You can do this later if there is a need.

4. Are you talking about diffs or binary deltas here? The first don't have this problem.

[AL|EN]

@lynx

1.That's why it's opt-in, I'm sure that when SR/IT maintainers will like to have such feature for those mods, a simple rearrange of the repo should not be a problem.

2. and 3. Yep, I can add this later, I'm asking because I've already receive such feature request from one player at beamdog thread.

4. Indeed for diffs it shouldn't be any problem. Binary deltas aren't something which I could achieve without will and direct cooperation with site admins.

[lynx]

So what kind of deltas are you talking about, just the fact that the mod changed, not actually any format? I don't see a problem there either; things under SCM will update nicely, while for normal archives you can wipe everything but inis and it should also be fine.

[AL|EN]

@lynx My main idea is to use git deltas.

After some experimentation, unfortunately it's impossible (or rather only possible if I force update to overwrite altered local mod files which is dangerous operation) for now to support "Delta-Updates" when multiple mods were extracted into the same top-level directory. So while all mods can be scanned and installed regardless of having extra top-level directories, they cannot receive delta-updates unless they have extra top-level directory. 

[cmorgan]

My apologies - can you throw a directory structure up that sorts this out for me? And we are talking about user folders/directories for mod archives, not user folders/directories within the game install folder, right?

[AL|EN]

@cmorgan Yes, user folders/directories for extracted mods. It has nothing to do with the way how anyone structure his mod. Everything is related to the situation when multiple mods which contain "Download=..." were extracted to the same top-level directory like this:

Quote

ModA
Download=http://github.com/UserA/ModA
D:\ExtractedMods\Quests\ModA
D:\ExtractedMods\Quests\Setup-ModA.tp2
D:\ExtractedMods\Quests\Setup-ModA.exe
ModB
Download=http://github.com/UserB/ModB
D:\ExtractedMods\Quests\ModB
D:\ExtractedMods\Quests\Setup-ModB.tp2
D:\ExtractedMods\Quests\Setup-ModB.exe

both mods has perfectly fine folder structures, they can be copied and installed without problems but they share top-level directory (Quests) so delta updates can't be preformed safely.

User will have to create one extra top-level directory like this:
 

Quote

D:\ExtractedMods\Quests\ModA\ModA\Items
D:\ExtractedMods\Quests\ModA\ModA\ModA.tp2
D:\ExtractedMods\Quests\ModA\ModA\Setup-ModA.exe
D:\ExtractedMods\Quests\ModB\ModB\Lib
D:\ExtractedMods\Quests\ModB\ModB\ModB.tp2
D:\ExtractedMods\Quests\ModB\ModA\Setup-ModB.exe

in order to be able to preform delta updates.

Edited December 31, 2020 by AL|EN
corrected example

[cmorgan]

Cool - and thank you. This is pertinent to me; I grep across an archive of mods when troubleshooting (both for my own modding purposes, looking for conflicting or overloaded states touched with I_C_T[1234], and for figuring out mod conflicts).  

We used to stop folks accidentally packaging mods up this way. Sounds like for my archive, it might be a good idea to do this. 

[AL|EN]

4 minutes ago, cmorgan said:

We used to stop folks accidentally packaging mods up this way. Sounds like for my archive, it might be a good idea to do this. 

Just to confirm: no changes are required for mods, even if they come with zip without extra top-level directory (good practice anyway). Users will simply create such extra top-level directory by themselves, one time only and all future updates will be handled correctly.

[lynx]

The download tool could do it for them quite easily.

Also for the deltas, I don't see a problem with shared directories. If two mods have the same filenames, there'll be problems much sooner than this. If you're nuking the dir before updating, then it's not a delta update anymore, but a redownload.

Dealing with user changes is tricky, but you could git stash beforehand and git stash pop afterwards. The only problem is if there are conflicts, since that requires a manual intervention.

[AL|EN]

Revisiting this topic, there is one thing I need to mention.

tl;dr
If you use GitHub releases to ship new versions, after creating a release do not edit it in any way. Yes, that includes manually replacing mod packages or manipulating tags. You can change the title and description.

If the mod is hosted on GitHub and you do not use releases, everything below doesn't apply to you but you are encouraged to read this anyway.

If the mod is hosted on GitHub and you use releases to ship new versions, there are some things you must not do because bad things will happen.

In order to not break things, simply follow the usual Git + GitHub workflow:

1. Update tp2 VERSION keyword, update readme with version, and changelog
2. Create final local commit with all 'ready-to-go' changes
3. Push local commits into a remote repository
4. Optionally, create mod packages that you will later add to the release or use automated tools
5. Create release via the GitHub web interface (or via tools) with the same tag as VERSION

This is the end of the road. After publishing release, you cannot:

• add other tags to the same release
• remove or replace existing tags
• replace mod packages with new ones that contain changes that were created and committed later than release tag

If you have done ANY modification to the local files, especially if you discover mistakes, and you want to 'update' the mod, you must repeat all of the steps above.

Helpful tools that follows required workflow:

There are new tools that will ensure that not only will save you time but ensure you will follow this workflow every time and save you from trivial mistakes.

Mod Release  - among other things, it prevents creating "2.0.0" release when your mod version is "1.8.0", prevents creating a new release if there are uncommitted file modifications and prevent creating a new release if the same release already exists.

Infinity Auto Packager - among other things, it prevents creating mod packages that might contain locally uncommitted changes

Clarification:

PI cannot use release packages for updating the mods, it uses release tags in order to match the state of local files with the state of remote files.

So especially you shouldn't replace mod packages inside release with new ones that contain changes. If you do, the mod update will not contain the latest changes and so the 'update' feature appears to be broken.

Forget about PI for one moment. Know that some smart people can use GitHub clients to 'update' all mods from GitHub by simply 'pulling latest tags'. They expect that the 'release tags' represent the code for the release because everything else that is hosted on GitHub (software/projects/etc) ensure this is true.

I hope this will clarify some aspects of mod updates via releases. Again, this doesn't concern you if your mod has no releases.

This is now included inside the documentation of Delta updates for mods.

Edited January 21, 2021 by AL|EN

[Magus]

"Delta" usually refers to bin files, I think the name is misleading.

Anyway, I don't feel enthusiastic about this. I think that any bugs coming up in a existing installation are better handled by CLUAConsole. There's just too much that can go wrong with a mid-game update.

But if you do actually implement this in one form or another, the first must have features are backup and rollback.

[AL|EN]

@Magus Well, how do you call an update that internally takes delta of the mod files and apply to local files in order to produce new file version without downloading it again?

I don't get the second part, can you clariffy?

[Magus]

It would be delta if you only downloaded the diffs (not repo or tag or release source) and then apply the diff to the local. But I think it's too much to take on. What you describe is, I don't know, git update, or incremental update? Well, it doesn't matter much, just the question of terminolgy.

The second part is quite simple - somewhere midgame I happily update some mods and suddenly hell breaks loose - some of them fail to install. Or install and cause crashes. Or screw up tkl refs. Or else.

I want to be sure that at any point I can revert to the original state and continue playing the savegames I have. Preferably, with one button click.

(Well, theoretically, since PI is windows-only and I don't have windows).

Edited January 21, 2021 by Magus

[AL|EN]

@Magus

But it has nothing to do with 'updating mods to latest mod release'. The thing which you describe is out of the scope of 'mod update' feature. Players can always revert to older mod version so the working state of the installation will be untouched. Also updating mods during the run is never a good idea.

The thing which you want is some sort of sophisticated multi-stage backup and restore feature. I'm afraid that this is too big to handle.